2023-11-15T04:12:25.843 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-15T04:12:25.844 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354 --verbose --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --owner scheduled_yuriw@teuthology --name yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354/orig.config.yaml 2023-11-15T04:12:25.899 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T04:12:25.992 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354 branch: wip-yuri4-testing-2023-11-13-0820-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7458354' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-11-13-0820-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 620f25c1794f62207424fc9319427943fb8d1fae ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 620f25c1794f62207424fc9319427943fb8d1fae selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-11-13-0820-pacific sha1: 620f25c1794f62207424fc9319427943fb8d1fae owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4886 sha1: 620f25c1794f62207424fc9319427943fb8d1fae sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-11-13-0820-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 620f25c1794f62207424fc9319427943fb8d1fae targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG0OvF9O3QqHLKEqxWfUUJCtLydCEOZRpVt7/ApFyU5En8GBhxgd5s/biYBzUuVRYD42he6oTvsZePaLDEtDN0E= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyyin6RZp5iH+Tz+eE48cVroDms3tb8Lb35sH8TCFBalDeVrCdON3Gnd5FWtFMXDAuV4CVCDw3t6Jg2nQInWDs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-14_20:45:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-15T04:12:25.993 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa; will attempt to use it 2023-11-15T04:12:25.994 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks 2023-11-15T04:12:25.994 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-15T04:12:25.996 INFO:teuthology.task.internal:Checking packages... 2023-11-15T04:12:26.020 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '620f25c1794f62207424fc9319427943fb8d1fae' 2023-11-15T04:12:26.021 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-15T04:12:26.021 INFO:teuthology.packaging:ref: None 2023-11-15T04:12:26.021 INFO:teuthology.packaging:tag: None 2023-11-15T04:12:26.022 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T04:12:26.022 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:12:26.022 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T04:12:26.259 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-170.g620f25c1 2023-11-15T04:12:26.260 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-15T04:12:26.267 INFO:teuthology.task.internal:no buildpackages task found 2023-11-15T04:12:26.267 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-15T04:12:26.294 INFO:teuthology.task.internal:Saving configuration 2023-11-15T04:12:26.313 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-15T04:12:26.320 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T04:12:26.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:07:27.471070', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG0OvF9O3QqHLKEqxWfUUJCtLydCEOZRpVt7/ApFyU5En8GBhxgd5s/biYBzUuVRYD42he6oTvsZePaLDEtDN0E='} 2023-11-15T04:12:26.372 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:07:27.469522', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyyin6RZp5iH+Tz+eE48cVroDms3tb8Lb35sH8TCFBalDeVrCdON3Gnd5FWtFMXDAuV4CVCDw3t6Jg2nQInWDs='} 2023-11-15T04:12:26.373 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-15T04:12:26.382 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-15T04:12:26.382 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-15T04:12:26.383 INFO:teuthology.run_tasks:Running task console_log... 2023-11-15T04:12:26.482 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f08d0eefb80>, signals=[15]) 2023-11-15T04:12:26.483 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-15T04:12:26.494 INFO:teuthology.task.internal:Opening connections... 2023-11-15T04:12:26.494 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2023-11-15T04:12:26.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:12:26.565 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2023-11-15T04:12:26.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:12:26.637 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-15T04:12:26.645 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2023-11-15T04:12:26.669 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2023-11-15T04:12:26.670 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2023-11-15T04:12:26.732 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="CentOS Stream" 2023-11-15T04:12:26.732 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="8" 2023-11-15T04:12:26.732 INFO:teuthology.orchestra.run.smithi047.stdout:ID="centos" 2023-11-15T04:12:26.733 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="rhel fedora" 2023-11-15T04:12:26.733 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="8" 2023-11-15T04:12:26.733 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el8" 2023-11-15T04:12:26.733 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T04:12:26.733 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2023-11-15T04:12:26.734 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T04:12:26.734 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://centos.org/" 2023-11-15T04:12:26.734 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T04:12:26.734 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T04:12:26.735 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T04:12:26.735 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2023-11-15T04:12:26.753 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2023-11-15T04:12:26.775 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2023-11-15T04:12:26.776 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2023-11-15T04:12:26.836 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2023-11-15T04:12:26.837 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="8" 2023-11-15T04:12:26.837 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2023-11-15T04:12:26.837 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2023-11-15T04:12:26.838 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="8" 2023-11-15T04:12:26.838 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el8" 2023-11-15T04:12:26.838 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T04:12:26.838 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2023-11-15T04:12:26.838 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T04:12:26.839 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2023-11-15T04:12:26.839 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T04:12:26.839 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T04:12:26.839 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T04:12:26.840 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2023-11-15T04:12:26.861 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-15T04:12:26.872 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-15T04:12:26.911 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-15T04:12:26.911 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2023-11-15T04:12:26.915 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2023-11-15T04:12:26.935 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-15T04:12:26.945 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-15T04:12:26.945 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2023-11-15T04:12:26.976 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2023-11-15T04:12:27.022 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-15T04:12:27.078 INFO:teuthology.run_tasks:Running task kernel... 2023-11-15T04:12:27.098 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-15T04:12:27.099 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-15T04:12:27.100 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T04:12:27.101 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T04:12:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:12:27.121 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2023-11-15T04:12:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:12:27.123 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2023-11-15T04:12:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:4.18.0-489.el8.x86_64 2023-11-15T04:12:27.182 INFO:teuthology.task.kernel:Running kernel on smithi190: 4.18.0-489.el8.x86_64 2023-11-15T04:12:27.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2023-11-15T04:12:27.185 INFO:teuthology.orchestra.run.smithi047.stdout:4.18.0-489.el8.x86_64 2023-11-15T04:12:27.185 INFO:teuthology.task.kernel:Running kernel on smithi047: 4.18.0-489.el8.x86_64 2023-11-15T04:12:27.185 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum install -y kernel 2023-11-15T04:12:31.501 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - AppStream 20 MB/s | 34 MB 00:01 2023-11-15T04:12:31.568 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - AppStream 20 MB/s | 34 MB 00:01 2023-11-15T04:12:39.218 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - BaseOS 35 MB/s | 53 MB 00:01 2023-11-15T04:12:39.410 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - BaseOS 41 MB/s | 53 MB 00:01 2023-11-15T04:12:47.217 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2023-11-15T04:12:47.726 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2023-11-15T04:12:47.935 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2023-11-15T04:12:48.036 INFO:teuthology.orchestra.run.smithi190.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-11-15T04:12:48.502 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-11-15T04:12:48.804 INFO:teuthology.orchestra.run.smithi047.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-15T04:12:49.395 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-11-15T04:12:50.773 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 16 MB 00:01 2023-11-15T04:12:53.237 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 292 kB/s | 24 kB 00:00 2023-11-15T04:12:54.780 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 480 kB/s | 24 kB 00:00 2023-11-15T04:12:56.869 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T04:12:56.870 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T04:12:57.138 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:12:57.139 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:12:57.139 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:12:57.139 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:12:57.139 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:12:57.140 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T04:12:57.140 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:12:57.140 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T04:12:57.140 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T04:12:57.140 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:12:57.141 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:12:57.141 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:12:57.141 INFO:teuthology.orchestra.run.smithi190.stdout:Install 3 Packages 2023-11-15T04:12:57.141 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:12:57.142 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 89 M 2023-11-15T04:12:57.142 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 96 M 2023-11-15T04:12:57.142 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:12:57.580 INFO:teuthology.orchestra.run.smithi190.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 25 MB/s | 10 MB 00:00 2023-11-15T04:12:58.304 INFO:teuthology.orchestra.run.smithi190.stdout:(2/3): kernel-core-4.18.0-521.el8.x86_64.rpm 39 MB/s | 43 MB 00:01 2023-11-15T04:12:58.662 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T04:12:58.663 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T04:12:58.805 INFO:teuthology.orchestra.run.smithi190.stdout:(3/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 22 MB/s | 36 MB 00:01 2023-11-15T04:12:58.806 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:12:58.806 INFO:teuthology.orchestra.run.smithi190.stdout:Total 54 MB/s | 89 MB 00:01 2023-11-15T04:12:58.961 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:12:58.961 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:12:58.962 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:12:58.962 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:12:58.962 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:12:58.962 INFO:teuthology.orchestra.run.smithi047.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T04:12:58.962 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:12:58.963 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T04:12:58.963 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T04:12:58.963 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:12:58.963 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:12:58.963 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:12:58.964 INFO:teuthology.orchestra.run.smithi047.stdout:Install 3 Packages 2023-11-15T04:12:58.964 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:12:58.964 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 89 M 2023-11-15T04:12:58.965 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 96 M 2023-11-15T04:12:58.965 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:12:59.381 INFO:teuthology.orchestra.run.smithi047.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 27 MB/s | 10 MB 00:00 2023-11-15T04:12:59.480 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:12:59.900 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:12:59.900 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:13:00.048 INFO:teuthology.orchestra.run.smithi047.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 34 MB/s | 36 MB 00:01 2023-11-15T04:13:00.648 INFO:teuthology.orchestra.run.smithi047.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 26 MB/s | 43 MB 00:01 2023-11-15T04:13:00.649 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:13:00.649 INFO:teuthology.orchestra.run.smithi047.stdout:Total 53 MB/s | 89 MB 00:01 2023-11-15T04:13:01.128 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:13:01.148 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:13:01.355 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:13:01.811 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:13:01.811 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:13:03.096 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:13:03.125 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:13:03.898 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:13:04.469 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:13:06.101 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:13:06.688 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:13:08.063 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:13:08.494 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:13:10.444 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:13:10.962 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:13:11.920 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:13:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:14.635 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:13:14.930 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:34.045 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:34.068 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:34.415 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:34.415 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:13:34.416 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:13:35.213 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:35.213 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:13:35.214 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:13:35.214 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T04:13:35.214 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T04:13:35.215 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:13:35.215 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:13:35.387 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:13:35.604 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:13:35.604 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:13:35.605 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T04:13:35.605 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T04:13:35.605 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T04:13:35.606 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-15T04:13:35.606 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T04:13:35.606 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:13:35.815 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:13:35.815 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:13:35.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2023-11-15T04:13:35.849 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2023-11-15T04:13:35.850 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2023-11-15T04:13:35.850 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.02 2023-11-15T04:13:35.850 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 129.el8 2023-11-15T04:13:35.850 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2023-11-15T04:13:35.850 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T04:13:35.851 INFO:teuthology.orchestra.run.smithi190.stdout:Group : System Environment/Base 2023-11-15T04:13:35.851 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 9265586 2023-11-15T04:13:35.851 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2023-11-15T04:13:35.851 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T04:13:35.851 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T04:13:35.852 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T04:13:35.852 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T04:13:35.852 INFO:teuthology.orchestra.run.smithi190.stdout:Relocations : (not relocatable) 2023-11-15T04:13:35.852 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : CentOS BuildSystem 2023-11-15T04:13:35.852 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2023-11-15T04:13:35.853 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T04:13:35.853 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2023-11-15T04:13:35.853 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2023-11-15T04:13:35.853 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:13:35.853 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T04:13:35.853 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T04:13:35.854 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T04:13:35.854 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2023-11-15T04:13:35.854 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:13:35.854 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T04:13:35.855 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T04:13:35.856 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 4.18.0-521.el8.x86_64 2023-11-15T04:13:35.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T04:13:37.733 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2023-11-15T04:13:38.330 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:38.355 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:38.747 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:38.748 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:13:38.748 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:13:39.881 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:13:39.881 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:13:39.881 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:13:39.881 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T04:13:39.882 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T04:13:39.882 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:13:39.882 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:13:40.031 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:13:40.053 INFO:teuthology.orchestra.run.smithi190.stderr:done 2023-11-15T04:13:40.055 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2023-11-15T04:13:40.079 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.79jgeQhm1k 2023-11-15T04:13:40.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.79jgeQhm1k 2023-11-15T04:13:40.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.79jgeQhm1k 2023-11-15T04:13:40.260 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:13:40.261 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:13:40.261 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T04:13:40.261 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T04:13:40.261 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T04:13:40.262 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-15T04:13:40.262 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T04:13:40.262 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:13:40.323 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.79jgeQhm1k is 6KB 2023-11-15T04:13:40.372 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.79jgeQhm1k 2023-11-15T04:13:40.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T04:13:40.473 INFO:teuthology.orchestra.run.smithi190.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T04:13:40.474 INFO:teuthology.orchestra.run.smithi190.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T04:13:40.474 INFO:teuthology.orchestra.run.smithi190.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T04:13:40.474 INFO:teuthology.orchestra.run.smithi190.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T04:13:40.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T04:13:40.479 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:13:40.480 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:13:40.481 DEBUG:teuthology.orchestra.run.smithi047:> sudo rpm -qi grub2-tools 2023-11-15T04:13:40.515 INFO:teuthology.orchestra.run.smithi047.stdout:Name : grub2-tools 2023-11-15T04:13:40.515 INFO:teuthology.orchestra.run.smithi047.stdout:Epoch : 1 2023-11-15T04:13:40.515 INFO:teuthology.orchestra.run.smithi047.stdout:Version : 2.02 2023-11-15T04:13:40.516 INFO:teuthology.orchestra.run.smithi047.stdout:Release : 129.el8 2023-11-15T04:13:40.516 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: x86_64 2023-11-15T04:13:40.516 INFO:teuthology.orchestra.run.smithi047.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T04:13:40.516 INFO:teuthology.orchestra.run.smithi047.stdout:Group : System Environment/Base 2023-11-15T04:13:40.516 INFO:teuthology.orchestra.run.smithi047.stdout:Size : 9265586 2023-11-15T04:13:40.517 INFO:teuthology.orchestra.run.smithi047.stdout:License : GPLv3+ 2023-11-15T04:13:40.517 INFO:teuthology.orchestra.run.smithi047.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T04:13:40.517 INFO:teuthology.orchestra.run.smithi047.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T04:13:40.517 INFO:teuthology.orchestra.run.smithi047.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T04:13:40.518 INFO:teuthology.orchestra.run.smithi047.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T04:13:40.518 INFO:teuthology.orchestra.run.smithi047.stdout:Relocations : (not relocatable) 2023-11-15T04:13:40.518 INFO:teuthology.orchestra.run.smithi047.stdout:Packager : CentOS BuildSystem 2023-11-15T04:13:40.518 INFO:teuthology.orchestra.run.smithi047.stdout:Vendor : CentOS 2023-11-15T04:13:40.518 INFO:teuthology.orchestra.run.smithi047.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T04:13:40.519 INFO:teuthology.orchestra.run.smithi047.stdout:Summary : Support tools for GRUB. 2023-11-15T04:13:40.519 INFO:teuthology.orchestra.run.smithi047.stdout:Description : 2023-11-15T04:13:40.519 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:13:40.519 INFO:teuthology.orchestra.run.smithi047.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T04:13:40.519 INFO:teuthology.orchestra.run.smithi047.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T04:13:40.520 INFO:teuthology.orchestra.run.smithi047.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T04:13:40.520 INFO:teuthology.orchestra.run.smithi047.stdout:hardware devices. 2023-11-15T04:13:40.520 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:13:40.520 INFO:teuthology.orchestra.run.smithi047.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T04:13:40.521 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T04:13:40.522 INFO:teuthology.task.kernel:Updating grub on smithi047 to boot 4.18.0-521.el8.x86_64 2023-11-15T04:13:40.522 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T04:13:40.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2023-11-15T04:13:42.079 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2023-11-15T04:13:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:done 2023-11-15T04:13:44.597 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2023-11-15T04:13:44.670 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.IdXVhvposk 2023-11-15T04:13:44.671 DEBUG:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IdXVhvposk 2023-11-15T04:13:44.742 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.IdXVhvposk 2023-11-15T04:13:44.883 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.IdXVhvposk is 6KB 2023-11-15T04:13:44.932 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.IdXVhvposk 2023-11-15T04:13:44.950 DEBUG:teuthology.orchestra.run.smithi047:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T04:13:45.018 INFO:teuthology.orchestra.run.smithi047.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T04:13:45.018 INFO:teuthology.orchestra.run.smithi047.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T04:13:45.019 INFO:teuthology.orchestra.run.smithi047.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T04:13:45.019 INFO:teuthology.orchestra.run.smithi047.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T04:13:45.020 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T04:13:45.183 DEBUG:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2023-11-15T04:14:10.650 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-15T04:14:10.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-15T04:14:10.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:14:15.190 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-15T04:14:15.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2023-11-15T04:14:15.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:15:10.706 DEBUG:teuthology.orchestra.remote:timed out 2023-11-15T04:15:15.194 DEBUG:teuthology.orchestra.remote:timed out 2023-11-15T04:15:19.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-15T04:15:19.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:15:20.027 DEBUG:teuthology.orchestra.run.smithi190:> true 2023-11-15T04:15:20.294 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-15T04:15:20.295 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-15T04:15:20.295 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2023-11-15T04:15:20.312 INFO:teuthology.orchestra.run.smithi190.stdout:4.18.0-521.el8.x86_64 2023-11-15T04:15:20.313 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T04:15:20.313 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T04:15:20.313 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T04:15:21.314 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-15T04:15:21.315 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T04:15:21.428 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2023-11-15T04:15:21.445 DEBUG:teuthology.parallel:result is None 2023-11-15T04:15:24.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2023-11-15T04:15:24.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:15:24.539 DEBUG:teuthology.orchestra.run.smithi047:> true 2023-11-15T04:15:24.801 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2023-11-15T04:15:24.802 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-15T04:15:24.802 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2023-11-15T04:15:24.822 INFO:teuthology.orchestra.run.smithi047.stdout:4.18.0-521.el8.x86_64 2023-11-15T04:15:24.823 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T04:15:24.823 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T04:15:24.824 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T04:15:25.825 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-15T04:15:25.825 DEBUG:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T04:15:25.901 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2023-11-15T04:15:25.917 DEBUG:teuthology.parallel:result is None 2023-11-15T04:15:25.918 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-15T04:15:25.925 INFO:teuthology.task.internal:Creating test directory... 2023-11-15T04:15:25.926 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T04:15:25.928 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T04:15:25.950 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-15T04:15:25.960 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-15T04:15:26.003 INFO:teuthology.task.internal:Creating archive directory... 2023-11-15T04:15:26.003 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T04:15:26.006 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T04:15:26.050 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-15T04:15:26.057 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-15T04:15:26.057 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T04:15:26.066 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T04:15:26.100 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:15:26.112 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:15:26.121 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:15:26.132 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:15:26.134 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-15T04:15:26.143 INFO:teuthology.task.internal:Configuring sudo... 2023-11-15T04:15:26.143 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T04:15:26.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T04:15:26.203 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-15T04:15:26.212 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-15T04:15:26.213 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T04:15:26.224 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T04:15:26.261 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:15:26.327 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:15:26.382 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:15:26.443 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:15:26.487 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:15:26.488 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T04:15:26.555 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:15:26.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:15:26.654 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:15:26.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:15:26.783 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-15T04:15:26.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T04:15:26.851 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2023-11-15T04:15:26.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2023-11-15T04:15:26.921 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:15:26.957 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:15:27.255 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-15T04:15:27.264 INFO:teuthology.task.internal:Starting timer... 2023-11-15T04:15:27.264 INFO:teuthology.run_tasks:Running task pcp... 2023-11-15T04:15:27.298 INFO:teuthology.run_tasks:Running task selinux... 2023-11-15T04:15:27.338 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-15T04:15:27.339 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2023-11-15T04:15:27.498 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: 2023-11-15T04:15:27.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2023-11-15T04:15:27.567 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2023-11-15T04:15:27.568 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-15T04:15:27.569 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2023-11-15T04:15:27.626 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2023-11-15T04:15:27.627 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2023-11-15T04:15:27.651 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2023-11-15T04:15:27.651 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi047.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2023-11-15T04:15:27.652 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:15:27.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:15:27.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:15:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:15:27.729 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-15T04:15:27.729 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2023-11-15T04:15:27.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2023-11-15T04:15:27.952 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-15T04:15:27.964 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-15T04:15:27.965 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-15T04:15:27.983 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-15T04:15:27.985 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-15T04:23:42.314 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2023-11-15T04:23:42.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2023-11-15T04:23:42.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:23:42.389 DEBUG:teuthology.orchestra.run.smithi047:> true 2023-11-15T04:23:42.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2023-11-15T04:23:42.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-15T04:23:42.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:23:42.491 DEBUG:teuthology.orchestra.run.smithi190:> true 2023-11-15T04:23:42.517 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-15T04:23:42.518 INFO:teuthology.run_tasks:Running task clock... 2023-11-15T04:23:42.532 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-15T04:23:42.532 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T04:23:42.533 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:23:42.536 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T04:23:42.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:23:42.574 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T04:23:42.589 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T04:23:42.590 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T04:23:42.603 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T04:23:42.660 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2023-11-15T04:23:42.668 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2023-11-15T04:23:42.673 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2023-11-15T04:23:42.679 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2023-11-15T04:23:42.690 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T04:23:42.693 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T04:23:42.706 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T04:23:42.706 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T04:23:42.781 INFO:teuthology.orchestra.run.smithi190.stderr:bash: ntpq: command not found 2023-11-15T04:23:42.784 INFO:teuthology.orchestra.run.smithi047.stderr:bash: ntpq: command not found 2023-11-15T04:23:42.784 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:23:42.785 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2023-11-15T04:23:42.785 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:23:42.785 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:23:42.785 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:23:42.786 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:23:42.788 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:23:42.788 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2023-11-15T04:23:42.788 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:23:42.788 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:23:42.789 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:23:42.789 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:23:42.790 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T04:23:42.800 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T04:23:42.801 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi047.front.sepia.ceph.com 2023-11-15T04:23:42.804 DEBUG:teuthology.orchestra.run.smithi047:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:23:42.804 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi190.front.sepia.ceph.com 2023-11-15T04:23:42.805 DEBUG:teuthology.orchestra.run.smithi190:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:23:43.713 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:24 ago on Wed 15 Nov 2023 04:21:19 AM UTC. 2023-11-15T04:23:43.853 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:43 ago on Wed 15 Nov 2023 04:22:00 AM UTC. 2023-11-15T04:23:45.830 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:23:45.831 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:23:45.831 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:23:45.832 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:23:45.832 INFO:teuthology.orchestra.run.smithi190.stdout:Resetting modules: 2023-11-15T04:23:45.832 INFO:teuthology.orchestra.run.smithi190.stdout: container-tools 2023-11-15T04:23:45.832 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:23:45.832 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:23:45.833 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:23:45.833 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:23:46.101 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:23:46.224 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:23:46.225 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:23:46.225 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:23:46.226 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:23:46.226 INFO:teuthology.orchestra.run.smithi047.stdout:Resetting modules: 2023-11-15T04:23:46.226 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools 2023-11-15T04:23:46.227 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:23:46.227 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:23:46.227 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:23:46.228 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:23:46.508 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:23:47.051 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:27 ago on Wed 15 Nov 2023 04:21:19 AM UTC. 2023-11-15T04:23:47.616 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:47 ago on Wed 15 Nov 2023 04:22:00 AM UTC. 2023-11-15T04:23:49.162 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:23:49.340 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:23:49.340 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-11-15T04:23:49.340 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:23:49.340 INFO:teuthology.orchestra.run.smithi190.stdout:Installing group/module packages: 2023-11-15T04:23:49.341 INFO:teuthology.orchestra.run.smithi190.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T04:23:49.341 INFO:teuthology.orchestra.run.smithi190.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T04:23:49.341 INFO:teuthology.orchestra.run.smithi190.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T04:23:49.341 INFO:teuthology.orchestra.run.smithi190.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T04:23:49.341 INFO:teuthology.orchestra.run.smithi190.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T04:23:49.342 INFO:teuthology.orchestra.run.smithi190.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T04:23:49.342 INFO:teuthology.orchestra.run.smithi190.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T04:23:49.342 INFO:teuthology.orchestra.run.smithi190.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T04:23:49.342 INFO:teuthology.orchestra.run.smithi190.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T04:23:49.342 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:23:49.343 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T04:23:49.343 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T04:23:49.343 INFO:teuthology.orchestra.run.smithi190.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T04:23:49.343 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2023-11-15T04:23:49.343 INFO:teuthology.orchestra.run.smithi190.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T04:23:49.344 INFO:teuthology.orchestra.run.smithi190.stdout:Installing module profiles: 2023-11-15T04:23:49.344 INFO:teuthology.orchestra.run.smithi190.stdout: container-tools/common 2023-11-15T04:23:49.344 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:23:49.344 INFO:teuthology.orchestra.run.smithi190.stdout:Enabling module streams: 2023-11-15T04:23:49.344 INFO:teuthology.orchestra.run.smithi190.stdout: container-tools rhel8 2023-11-15T04:23:49.344 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:23:49.345 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:23:49.345 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:23:49.345 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2023-11-15T04:23:49.345 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:23:49.346 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 27 M 2023-11-15T04:23:49.346 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 89 M 2023-11-15T04:23:49.346 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:23:49.812 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.3 MB/s | 1.0 MB 00:00 2023-11-15T04:23:49.837 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): cockpit-podman-78-1.module_el8+712+4cd1 1.6 MB/s | 750 kB 00:00 2023-11-15T04:23:49.904 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 411 kB/s | 38 kB 00:00 2023-11-15T04:23:49.930 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.6 MB/s | 241 kB 00:00 2023-11-15T04:23:49.966 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:23:50.021 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.9 MB/s | 178 kB 00:00 2023-11-15T04:23:50.155 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 12 MB/s | 9.6 MB 00:00 2023-11-15T04:23:50.165 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:23:50.166 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2023-11-15T04:23:50.166 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:23:50.166 INFO:teuthology.orchestra.run.smithi047.stdout:Installing group/module packages: 2023-11-15T04:23:50.166 INFO:teuthology.orchestra.run.smithi047.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T04:23:50.167 INFO:teuthology.orchestra.run.smithi047.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T04:23:50.167 INFO:teuthology.orchestra.run.smithi047.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T04:23:50.167 INFO:teuthology.orchestra.run.smithi047.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T04:23:50.167 INFO:teuthology.orchestra.run.smithi047.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T04:23:50.168 INFO:teuthology.orchestra.run.smithi047.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T04:23:50.168 INFO:teuthology.orchestra.run.smithi047.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T04:23:50.168 INFO:teuthology.orchestra.run.smithi047.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T04:23:50.168 INFO:teuthology.orchestra.run.smithi047.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T04:23:50.168 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:23:50.169 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T04:23:50.169 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T04:23:50.169 INFO:teuthology.orchestra.run.smithi047.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T04:23:50.169 INFO:teuthology.orchestra.run.smithi047.stdout:Installing weak dependencies: 2023-11-15T04:23:50.170 INFO:teuthology.orchestra.run.smithi047.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T04:23:50.170 INFO:teuthology.orchestra.run.smithi047.stdout:Installing module profiles: 2023-11-15T04:23:50.170 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools/common 2023-11-15T04:23:50.170 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:23:50.170 INFO:teuthology.orchestra.run.smithi047.stdout:Enabling module streams: 2023-11-15T04:23:50.171 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools rhel8 2023-11-15T04:23:50.171 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:23:50.171 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:23:50.171 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:23:50.172 INFO:teuthology.orchestra.run.smithi047.stdout:Install 13 Packages 2023-11-15T04:23:50.172 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:23:50.173 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 27 M 2023-11-15T04:23:50.173 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 89 M 2023-11-15T04:23:50.173 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:23:50.180 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 156 kB/s | 25 kB 00:00 2023-11-15T04:23:50.239 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 12 MB/s | 3.9 MB 00:00 2023-11-15T04:23:50.256 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 935 kB/s | 94 kB 00:00 2023-11-15T04:23:50.323 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 787 kB/s | 53 kB 00:00 2023-11-15T04:23:50.365 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 20 MB/s | 2.5 MB 00:00 2023-11-15T04:23:50.390 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 612 kB/s | 41 kB 00:00 2023-11-15T04:23:50.574 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 21 MB/s | 8.1 MB 00:00 2023-11-15T04:23:50.575 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:23:50.575 INFO:teuthology.orchestra.run.smithi190.stdout:Total 22 MB/s | 27 MB 00:01 2023-11-15T04:23:50.648 INFO:teuthology.orchestra.run.smithi047.stdout:(1/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.3 MB/s | 1.0 MB 00:00 2023-11-15T04:23:50.673 INFO:teuthology.orchestra.run.smithi047.stdout:(2/13): cockpit-podman-78-1.module_el8+712+4cd1 1.6 MB/s | 750 kB 00:00 2023-11-15T04:23:50.724 INFO:teuthology.orchestra.run.smithi047.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 505 kB/s | 38 kB 00:00 2023-11-15T04:23:50.757 INFO:teuthology.orchestra.run.smithi047.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.8 MB/s | 241 kB 00:00 2023-11-15T04:23:50.841 INFO:teuthology.orchestra.run.smithi047.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.1 MB/s | 178 kB 00:00 2023-11-15T04:23:50.843 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:23:50.865 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:23:50.865 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:23:51.008 INFO:teuthology.orchestra.run.smithi047.stdout:(6/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 12 MB/s | 9.6 MB 00:00 2023-11-15T04:23:51.033 INFO:teuthology.orchestra.run.smithi047.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 129 kB/s | 25 kB 00:00 2023-11-15T04:23:51.100 INFO:teuthology.orchestra.run.smithi047.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 10 MB/s | 3.9 MB 00:00 2023-11-15T04:23:51.121 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:23:51.126 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:23:51.127 INFO:teuthology.orchestra.run.smithi047.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 803 kB/s | 94 kB 00:00 2023-11-15T04:23:51.192 INFO:teuthology.orchestra.run.smithi047.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 790 kB/s | 53 kB 00:00 2023-11-15T04:23:51.235 INFO:teuthology.orchestra.run.smithi047.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 19 MB/s | 2.5 MB 00:00 2023-11-15T04:23:51.260 INFO:teuthology.orchestra.run.smithi047.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 610 kB/s | 41 kB 00:00 2023-11-15T04:23:51.427 INFO:teuthology.orchestra.run.smithi047.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 21 MB/s | 8.1 MB 00:00 2023-11-15T04:23:51.428 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:23:51.428 INFO:teuthology.orchestra.run.smithi047.stdout:Total 21 MB/s | 27 MB 00:01 2023-11-15T04:23:51.720 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:23:51.744 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:23:51.744 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:23:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:23:52.025 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:23:52.029 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:23:52.046 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T04:23:52.217 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T04:23:52.419 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T04:23:52.675 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:23:52.739 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:23:52.988 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T04:23:53.189 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T04:23:53.199 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T04:23:53.394 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T04:23:53.633 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:23:54.183 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:23:54.216 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T04:23:54.405 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T04:23:54.644 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T04:23:54.904 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T04:23:55.263 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:23:55.294 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T04:23:55.494 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T04:23:55.712 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T04:23:55.961 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T04:23:56.183 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:23:56.384 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T04:23:56.396 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T04:23:56.616 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:23:57.038 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:23:57.039 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T04:23:57.039 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T04:23:57.039 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T04:23:57.039 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:23:57.040 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T04:23:57.040 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:23:57.040 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T04:23:57.040 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T04:23:57.040 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T04:23:57.040 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T04:23:57.041 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:23:57.041 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T04:23:57.335 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:23:57.573 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T04:23:57.762 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:23:57.836 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T04:23:57.836 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:23:57.836 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:23:57.836 INFO:teuthology.orchestra.run.smithi190.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:57.837 INFO:teuthology.orchestra.run.smithi190.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:57.837 INFO:teuthology.orchestra.run.smithi190.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:23:57.837 INFO:teuthology.orchestra.run.smithi190.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T04:23:57.837 INFO:teuthology.orchestra.run.smithi190.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:57.837 INFO:teuthology.orchestra.run.smithi190.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:57.838 INFO:teuthology.orchestra.run.smithi190.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:23:57.838 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T04:23:57.838 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T04:23:57.838 INFO:teuthology.orchestra.run.smithi190.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:57.839 INFO:teuthology.orchestra.run.smithi190.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:57.839 INFO:teuthology.orchestra.run.smithi190.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T04:23:57.839 INFO:teuthology.orchestra.run.smithi190.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T04:23:57.839 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:23:57.839 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:23:58.080 DEBUG:teuthology.parallel:result is None 2023-11-15T04:23:58.248 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:23:58.248 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T04:23:58.249 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T04:23:58.249 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T04:23:58.249 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:23:58.250 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T04:23:58.250 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:23:58.250 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T04:23:58.251 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T04:23:58.251 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T04:23:58.251 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T04:23:58.251 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:23:58.252 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T04:23:59.088 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T04:23:59.089 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:23:59.089 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:23:59.089 INFO:teuthology.orchestra.run.smithi047.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:59.090 INFO:teuthology.orchestra.run.smithi047.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:59.090 INFO:teuthology.orchestra.run.smithi047.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:23:59.090 INFO:teuthology.orchestra.run.smithi047.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T04:23:59.091 INFO:teuthology.orchestra.run.smithi047.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:59.091 INFO:teuthology.orchestra.run.smithi047.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:59.091 INFO:teuthology.orchestra.run.smithi047.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:23:59.092 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T04:23:59.092 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T04:23:59.092 INFO:teuthology.orchestra.run.smithi047.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:59.093 INFO:teuthology.orchestra.run.smithi047.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:23:59.093 INFO:teuthology.orchestra.run.smithi047.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T04:23:59.094 INFO:teuthology.orchestra.run.smithi047.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T04:23:59.094 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:23:59.094 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:23:59.378 DEBUG:teuthology.parallel:result is None 2023-11-15T04:23:59.379 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T04:23:59.388 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T04:23:59.389 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi047.front.sepia.ceph.com 2023-11-15T04:23:59.389 DEBUG:teuthology.orchestra.run.smithi047:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:23:59.390 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi190.front.sepia.ceph.com 2023-11-15T04:23:59.390 DEBUG:teuthology.orchestra.run.smithi190:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:23:59.545 DEBUG:teuthology.parallel:result is None 2023-11-15T04:23:59.588 DEBUG:teuthology.parallel:result is None 2023-11-15T04:23:59.589 INFO:teuthology.run_tasks:Running task install... 2023-11-15T04:23:59.598 DEBUG:teuthology.task.install:project ceph 2023-11-15T04:23:59.599 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'}} 2023-11-15T04:23:59.599 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T04:23:59.599 INFO:teuthology.task.install:Using flavor: default 2023-11-15T04:23:59.606 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-15T04:23:59.607 INFO:teuthology.task.install:extra packages: [] 2023-11-15T04:23:59.607 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T04:23:59.608 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:23:59.609 INFO:teuthology.packaging:ref: None 2023-11-15T04:23:59.609 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:23:59.609 INFO:teuthology.packaging:branch: None 2023-11-15T04:23:59.610 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:23:59.818 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:23:59.819 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:23:59.822 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T04:23:59.822 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:23:59.822 INFO:teuthology.packaging:ref: None 2023-11-15T04:23:59.823 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:23:59.823 INFO:teuthology.packaging:branch: None 2023-11-15T04:23:59.823 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:24:00.020 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:24:00.021 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:24:00.247 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T04:24:00.248 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T04:24:00.265 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T04:24:00.265 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T04:24:00.310 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T04:24:00.311 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-15T04:24:00.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T04:24:00.317 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T04:24:00.318 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:24:00.318 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T04:24:00.344 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T04:24:00.344 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:24:00.344 INFO:teuthology.packaging:ref: None 2023-11-15T04:24:00.345 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:24:00.345 INFO:teuthology.packaging:branch: None 2023-11-15T04:24:00.345 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:24:00.345 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T04:24:00.354 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T04:24:00.355 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:24:00.355 INFO:teuthology.packaging:ref: None 2023-11-15T04:24:00.355 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:24:00.355 INFO:teuthology.packaging:branch: None 2023-11-15T04:24:00.355 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:24:00.356 DEBUG:teuthology.orchestra.run.smithi047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T04:24:00.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T04:24:00.431 DEBUG:teuthology.orchestra.run.smithi047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T04:24:00.496 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T04:24:00.520 DEBUG:teuthology.orchestra.run.smithi047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T04:24:00.555 INFO:teuthology.orchestra.run.smithi047.stdout:check_obsoletes = 1 2023-11-15T04:24:00.555 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum clean all 2023-11-15T04:24:00.566 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2023-11-15T04:24:00.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2023-11-15T04:24:00.842 INFO:teuthology.orchestra.run.smithi190.stdout:76 files removed 2023-11-15T04:24:00.865 INFO:teuthology.orchestra.run.smithi047.stdout:76 files removed 2023-11-15T04:24:00.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2023-11-15T04:24:00.902 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw 2023-11-15T04:24:01.358 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 933 kB/s | 74 kB 00:00 2023-11-15T04:24:01.453 INFO:teuthology.orchestra.run.smithi047.stdout:ceph packages for x86_64 883 kB/s | 74 kB 00:00 2023-11-15T04:24:01.476 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 190 kB/s | 15 kB 00:00 2023-11-15T04:24:01.579 INFO:teuthology.orchestra.run.smithi047.stdout:ceph noarch packages 184 kB/s | 15 kB 00:00 2023-11-15T04:24:01.585 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2023-11-15T04:24:01.688 INFO:teuthology.orchestra.run.smithi047.stdout:ceph source packages 23 kB/s | 1.7 kB 00:00 2023-11-15T04:24:03.694 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - AppStream 16 MB/s | 34 MB 00:02 2023-11-15T04:24:04.029 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - AppStream 15 MB/s | 34 MB 00:02 2023-11-15T04:24:11.228 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - BaseOS 41 MB/s | 53 MB 00:01 2023-11-15T04:24:12.130 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - BaseOS 43 MB/s | 53 MB 00:01 2023-11-15T04:24:19.545 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - Extras 28 kB/s | 18 kB 00:00 2023-11-15T04:24:20.261 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-11-15T04:24:20.862 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - PowerTools 15 MB/s | 6.6 MB 00:00 2023-11-15T04:24:21.130 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2023-11-15T04:24:21.739 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-11-15T04:24:22.281 INFO:teuthology.orchestra.run.smithi190.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-11-15T04:24:22.557 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 6.6 MB 00:00 2023-11-15T04:24:23.298 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2023-11-15T04:24:24.135 INFO:teuthology.orchestra.run.smithi047.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-11-15T04:24:26.196 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 16 MB 00:01 2023-11-15T04:24:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 311 kB/s | 24 kB 00:00 2023-11-15T04:24:30.572 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 414 kB/s | 24 kB 00:00 2023-11-15T04:24:31.831 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:24:31.833 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:24:31.834 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-11-15T04:24:31.834 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:24:31.834 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:24:31.834 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T04:24:31.835 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2023-11-15T04:24:31.835 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:24:31.835 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T04:24:31.835 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:24:31.835 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T04:24:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T04:24:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T04:24:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T04:24:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T04:24:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T04:24:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T04:24:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T04:24:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T04:24:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T04:24:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T04:24:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T04:24:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T04:24:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T04:24:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T04:24:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T04:24:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T04:24:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T04:24:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T04:24:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T04:24:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T04:24:31.840 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T04:24:31.840 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T04:24:31.841 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T04:24:31.841 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:24:31.841 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:24:31.841 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:24:31.842 INFO:teuthology.orchestra.run.smithi190.stdout:Install 25 Packages 2023-11-15T04:24:31.842 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2023-11-15T04:24:31.842 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:24:31.843 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 59 M 2023-11-15T04:24:31.843 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:24:32.234 INFO:teuthology.orchestra.run.smithi190.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 44 MB/s | 6.0 MB 00:00 2023-11-15T04:24:32.259 INFO:teuthology.orchestra.run.smithi190.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-11-15T04:24:32.292 INFO:teuthology.orchestra.run.smithi190.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2023-11-15T04:24:32.335 INFO:teuthology.orchestra.run.smithi190.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-11-15T04:24:32.426 INFO:teuthology.orchestra.run.smithi190.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2023-11-15T04:24:32.460 INFO:teuthology.orchestra.run.smithi190.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.3 MB/s | 44 kB 00:00 2023-11-15T04:24:32.485 INFO:teuthology.orchestra.run.smithi190.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T04:24:32.511 INFO:teuthology.orchestra.run.smithi190.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2023-11-15T04:24:32.546 INFO:teuthology.orchestra.run.smithi190.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T04:24:32.569 INFO:teuthology.orchestra.run.smithi190.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2023-11-15T04:24:32.595 INFO:teuthology.orchestra.run.smithi190.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T04:24:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2023-11-15T04:24:32.996 INFO:teuthology.orchestra.run.smithi190.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 27 MB/s | 24 MB 00:00 2023-11-15T04:24:33.326 INFO:teuthology.orchestra.run.smithi190.stdout:(14/27): librdkafka-0.11.4-3.el8.x86_64.rpm 598 kB/s | 354 kB 00:00 2023-11-15T04:24:33.346 INFO:teuthology.orchestra.run.smithi190.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 742 kB/s | 259 kB 00:00 2023-11-15T04:24:33.388 INFO:teuthology.orchestra.run.smithi190.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 656 kB/s | 44 kB 00:00 2023-11-15T04:24:33.430 INFO:teuthology.orchestra.run.smithi190.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 505 kB/s | 42 kB 00:00 2023-11-15T04:24:33.464 INFO:teuthology.orchestra.run.smithi190.stdout:(18/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 625 kB/s | 47 kB 00:00 2023-11-15T04:24:33.489 INFO:teuthology.orchestra.run.smithi190.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.3 MB/s | 80 kB 00:00 2023-11-15T04:24:33.615 INFO:teuthology.orchestra.run.smithi190.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 773 kB/s | 116 kB 00:00 2023-11-15T04:24:33.690 INFO:teuthology.orchestra.run.smithi190.stdout:(21/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.5 MB/s | 306 kB 00:00 2023-11-15T04:24:33.715 INFO:teuthology.orchestra.run.smithi190.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.8 MB/s | 181 kB 00:00 2023-11-15T04:24:33.740 INFO:teuthology.orchestra.run.smithi190.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2023-11-15T04:24:33.774 INFO:teuthology.orchestra.run.smithi190.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-11-15T04:24:33.849 INFO:teuthology.orchestra.run.smithi190.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2023-11-15T04:24:33.917 INFO:teuthology.orchestra.run.smithi190.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 29 MB/s | 4.0 MB 00:00 2023-11-15T04:24:35.041 INFO:teuthology.orchestra.run.smithi190.stdout:(27/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 59 kB/s | 145 kB 00:02 2023-11-15T04:24:35.043 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:24:35.044 INFO:teuthology.orchestra.run.smithi190.stdout:Total 18 MB/s | 59 MB 00:03 2023-11-15T04:24:35.111 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:24:35.161 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:24:35.161 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:24:35.494 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:24:35.497 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:24:35.589 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:24:35.593 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:24:35.593 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2023-11-15T04:24:35.593 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:24:35.594 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:24:35.594 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T04:24:35.594 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2023-11-15T04:24:35.595 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:24:35.595 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T04:24:35.595 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:24:35.595 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T04:24:35.595 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T04:24:35.596 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T04:24:35.596 INFO:teuthology.orchestra.run.smithi047.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T04:24:35.596 INFO:teuthology.orchestra.run.smithi047.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T04:24:35.597 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T04:24:35.597 INFO:teuthology.orchestra.run.smithi047.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T04:24:35.597 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T04:24:35.597 INFO:teuthology.orchestra.run.smithi047.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T04:24:35.598 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T04:24:35.598 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T04:24:35.598 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T04:24:35.598 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T04:24:35.599 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T04:24:35.599 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T04:24:35.599 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T04:24:35.599 INFO:teuthology.orchestra.run.smithi047.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T04:24:35.600 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T04:24:35.600 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T04:24:35.600 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T04:24:35.600 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T04:24:35.600 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T04:24:35.601 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T04:24:35.601 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T04:24:35.601 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:24:35.602 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:24:35.602 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:24:35.602 INFO:teuthology.orchestra.run.smithi047.stdout:Install 25 Packages 2023-11-15T04:24:35.602 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 2 Packages 2023-11-15T04:24:35.603 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:24:35.603 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 59 M 2023-11-15T04:24:35.604 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:24:36.170 INFO:teuthology.orchestra.run.smithi047.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 38 MB/s | 6.0 MB 00:00 2023-11-15T04:24:36.195 INFO:teuthology.orchestra.run.smithi047.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 995 kB/s | 25 kB 00:00 2023-11-15T04:24:36.237 INFO:teuthology.orchestra.run.smithi047.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-11-15T04:24:36.291 INFO:teuthology.orchestra.run.smithi047.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 8.8 MB/s | 486 kB 00:00 2023-11-15T04:24:36.404 INFO:teuthology.orchestra.run.smithi047.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 36 MB/s | 4.1 MB 00:00 2023-11-15T04:24:36.430 INFO:teuthology.orchestra.run.smithi047.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-15T04:24:36.455 INFO:teuthology.orchestra.run.smithi047.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T04:24:36.480 INFO:teuthology.orchestra.run.smithi047.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-15T04:24:36.514 INFO:teuthology.orchestra.run.smithi047.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T04:24:36.548 INFO:teuthology.orchestra.run.smithi047.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-15T04:24:36.573 INFO:teuthology.orchestra.run.smithi047.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T04:24:36.715 INFO:teuthology.orchestra.run.smithi047.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-15T04:24:36.766 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:24:36.998 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T04:24:37.016 INFO:teuthology.orchestra.run.smithi047.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:01 2023-11-15T04:24:37.041 INFO:teuthology.orchestra.run.smithi047.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 311 kB/s | 145 kB 00:00 2023-11-15T04:24:37.177 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T04:24:37.234 INFO:teuthology.orchestra.run.smithi047.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 227 kB/s | 44 kB 00:00 2023-11-15T04:24:37.266 INFO:teuthology.orchestra.run.smithi047.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 642 kB/s | 354 kB 00:00 2023-11-15T04:24:37.352 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:24:37.358 INFO:teuthology.orchestra.run.smithi047.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 513 kB/s | 47 kB 00:00 2023-11-15T04:24:37.383 INFO:teuthology.orchestra.run.smithi047.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 283 kB/s | 42 kB 00:00 2023-11-15T04:24:37.426 INFO:teuthology.orchestra.run.smithi047.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.2 MB/s | 80 kB 00:00 2023-11-15T04:24:37.467 INFO:teuthology.orchestra.run.smithi047.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 575 kB/s | 259 kB 00:00 2023-11-15T04:24:37.518 INFO:teuthology.orchestra.run.smithi047.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 868 kB/s | 116 kB 00:00 2023-11-15T04:24:37.560 INFO:teuthology.orchestra.run.smithi047.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-11-15T04:24:37.602 INFO:teuthology.orchestra.run.smithi047.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2023-11-15T04:24:37.627 INFO:teuthology.orchestra.run.smithi047.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.5 MB/s | 306 kB 00:00 2023-11-15T04:24:37.652 INFO:teuthology.orchestra.run.smithi047.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 814 kB/s | 75 kB 00:00 2023-11-15T04:24:37.752 INFO:teuthology.orchestra.run.smithi047.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2023-11-15T04:24:37.762 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:24:37.811 INFO:teuthology.orchestra.run.smithi047.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 22 MB/s | 4.0 MB 00:00 2023-11-15T04:24:37.813 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:24:37.814 INFO:teuthology.orchestra.run.smithi047.stdout:Total 26 MB/s | 59 MB 00:02 2023-11-15T04:24:37.885 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:24:37.940 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:24:37.940 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:24:37.952 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:24:38.041 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:24:38.284 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:24:38.315 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:24:38.319 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:24:38.444 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:24:38.512 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:24:38.692 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T04:24:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:24:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:24:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:24:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:24:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:24:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T04:24:39.740 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T04:24:39.909 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:24:39.960 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:24:39.966 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T04:24:40.148 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:24:40.166 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:24:40.310 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:24:40.485 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:24:40.562 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:24:40.593 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:24:40.806 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:24:40.908 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:24:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:24:41.174 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:24:41.347 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:24:41.389 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:24:41.458 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:24:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:24:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:24:41.667 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T04:24:41.822 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T04:24:41.848 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:24:42.061 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T04:24:42.201 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:24:42.272 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:24:42.304 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:24:42.468 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T04:24:42.631 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:24:42.676 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T04:24:42.709 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:24:42.866 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T04:24:42.887 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T04:24:43.085 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T04:24:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:24:43.157 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:24:43.330 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:24:43.509 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:24:43.699 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:24:43.796 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:24:43.956 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:24:44.428 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:24:44.656 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:24:44.749 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:24:44.936 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T04:24:45.154 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T04:24:45.314 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:24:45.386 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:24:45.619 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T04:24:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:24:45.815 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T04:24:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:24:46.004 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T04:24:46.206 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T04:24:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:24:46.878 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:24:46.984 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:24:47.167 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:24:49.012 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:24:49.247 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:24:50.025 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:24:50.304 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:24:50.407 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:24:50.572 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:25:05.665 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:25:05.666 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2023-11-15T04:25:05.666 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2023-11-15T04:25:05.666 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /dev 2023-11-15T04:25:05.666 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /run 2023-11-15T04:25:05.667 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2023-11-15T04:25:05.667 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2023-11-15T04:25:05.667 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2023-11-15T04:25:05.667 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2023-11-15T04:25:05.667 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /dev 2023-11-15T04:25:05.667 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:06.999 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:25:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:25:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:25:07.333 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:25:07.334 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:25:07.994 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:25:07.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T04:25:07.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T04:25:07.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T04:25:07.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:25:07.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:25:07.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:25:07.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T04:25:07.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T04:25:07.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:25:07.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T04:25:07.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T04:25:07.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:25:07.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:25:07.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T04:25:07.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T04:25:07.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T04:25:07.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T04:25:07.998 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T04:25:07.998 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T04:25:07.998 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T04:25:07.998 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T04:25:07.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T04:25:07.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T04:25:07.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T04:25:07.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T04:25:07.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:25:08.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T04:25:08.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T04:25:09.145 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:25:09.145 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:09.145 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2023-11-15T04:25:09.146 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.146 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:25:09.146 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.146 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.146 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.147 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.147 INFO:teuthology.orchestra.run.smithi190.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T04:25:09.147 INFO:teuthology.orchestra.run.smithi190.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T04:25:09.147 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T04:25:09.147 INFO:teuthology.orchestra.run.smithi190.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T04:25:09.148 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.148 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T04:25:09.148 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T04:25:09.148 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T04:25:09.148 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.149 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T04:25:09.149 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.149 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T04:25:09.149 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T04:25:09.149 INFO:teuthology.orchestra.run.smithi190.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T04:25:09.149 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.150 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.150 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.150 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T04:25:09.150 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.150 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.151 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:09.153 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:09.153 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:25:09.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2023-11-15T04:25:10.305 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:43 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:25:11.606 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:25:11.606 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /sys 2023-11-15T04:25:11.607 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /proc 2023-11-15T04:25:11.607 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /dev 2023-11-15T04:25:11.607 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /run 2023-11-15T04:25:11.607 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /mnt 2023-11-15T04:25:11.607 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /var/tmp 2023-11-15T04:25:11.608 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /home 2023-11-15T04:25:11.608 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /tmp 2023-11-15T04:25:11.608 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /dev 2023-11-15T04:25:11.608 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:11.977 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:25:11.977 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:11.978 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:25:11.978 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:11.978 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:25:11.978 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T04:25:11.979 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:25:11.979 INFO:teuthology.orchestra.run.smithi190.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T04:25:11.979 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T04:25:11.979 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T04:25:11.979 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T04:25:11.980 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T04:25:11.980 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:11.980 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:25:11.980 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:11.980 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2023-11-15T04:25:11.980 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:11.981 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 53 M 2023-11-15T04:25:11.981 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 229 M 2023-11-15T04:25:11.982 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:25:12.396 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.1 MB/s | 169 kB 00:00 2023-11-15T04:25:12.937 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 74 MB/s | 52 MB 00:00 2023-11-15T04:25:12.962 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 281 kB/s | 203 kB 00:00 2023-11-15T04:25:12.987 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 317 kB/s | 187 kB 00:00 2023-11-15T04:25:13.054 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 770 kB/s | 71 kB 00:00 2023-11-15T04:25:13.084 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:25:13.168 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:25:13.168 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:25:13.313 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 861 kB/s | 323 kB 00:00 2023-11-15T04:25:13.314 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:25:13.314 INFO:teuthology.orchestra.run.smithi190.stdout:Total 40 MB/s | 53 MB 00:01 2023-11-15T04:25:13.356 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:25:13.377 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:25:13.377 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:25:13.405 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:25:13.406 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:25:13.695 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:25:13.698 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:25:14.177 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:25:14.181 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:25:14.182 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T04:25:14.182 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T04:25:14.182 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T04:25:14.182 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:25:14.182 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:25:14.183 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:25:14.183 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T04:25:14.183 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T04:25:14.183 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:25:14.183 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T04:25:14.184 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T04:25:14.184 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:25:14.184 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:25:14.186 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T04:25:14.186 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T04:25:14.186 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T04:25:14.186 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T04:25:14.187 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T04:25:14.187 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T04:25:14.187 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T04:25:14.187 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T04:25:14.188 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T04:25:14.188 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T04:25:14.188 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T04:25:14.188 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T04:25:14.188 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:25:14.189 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T04:25:14.189 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T04:25:14.415 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T04:25:14.598 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T04:25:14.771 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:25:14.850 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:25:15.061 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T04:25:15.204 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:25:15.571 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:25:15.571 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:15.571 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2023-11-15T04:25:15.572 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.572 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:25:15.572 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.572 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.573 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.573 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.573 INFO:teuthology.orchestra.run.smithi047.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T04:25:15.573 INFO:teuthology.orchestra.run.smithi047.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T04:25:15.573 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T04:25:15.573 INFO:teuthology.orchestra.run.smithi047.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T04:25:15.574 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.574 INFO:teuthology.orchestra.run.smithi047.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T04:25:15.574 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T04:25:15.574 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T04:25:15.574 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.575 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T04:25:15.575 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.575 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T04:25:15.575 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T04:25:15.576 INFO:teuthology.orchestra.run.smithi047.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T04:25:15.576 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.576 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.576 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.576 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T04:25:15.577 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.577 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.577 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:15.577 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:15.577 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:25:15.734 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-test 2023-11-15T04:25:16.726 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:46 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:25:18.584 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:25:18.585 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:18.586 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:25:18.586 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:18.586 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:25:18.586 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T04:25:18.586 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:25:18.587 INFO:teuthology.orchestra.run.smithi047.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T04:25:18.587 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T04:25:18.587 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T04:25:18.587 INFO:teuthology.orchestra.run.smithi047.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T04:25:18.587 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T04:25:18.588 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:18.588 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:25:18.588 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:18.588 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2023-11-15T04:25:18.588 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:18.589 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 53 M 2023-11-15T04:25:18.589 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 229 M 2023-11-15T04:25:18.590 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:25:18.681 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 169 kB 00:00 2023-11-15T04:25:19.406 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 67 MB/s | 52 MB 00:00 2023-11-15T04:25:19.432 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 252 kB/s | 203 kB 00:00 2023-11-15T04:25:19.448 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 244 kB/s | 187 kB 00:00 2023-11-15T04:25:19.524 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 770 kB/s | 71 kB 00:00 2023-11-15T04:25:19.840 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 745 kB/s | 323 kB 00:00 2023-11-15T04:25:19.842 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:25:19.842 INFO:teuthology.orchestra.run.smithi047.stdout:Total 42 MB/s | 53 MB 00:01 2023-11-15T04:25:19.889 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:25:19.913 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:25:19.913 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:25:20.049 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:25:20.251 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:25:20.254 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:25:20.319 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:25:20.607 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:25:20.607 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T04:25:20.607 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T04:25:20.607 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T04:25:20.608 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T04:25:20.608 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T04:25:20.761 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:25:21.010 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T04:25:21.233 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T04:25:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T04:25:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:21.323 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:25:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T04:25:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T04:25:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T04:25:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:21.324 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:25:21.434 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:25:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:25:21.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2023-11-15T04:25:21.710 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T04:25:21.892 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:25:22.434 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:55 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:25:24.105 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:25:24.108 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:24.108 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-11-15T04:25:24.108 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:24.109 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:25:24.109 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T04:25:24.109 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:25:24.109 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T04:25:24.109 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T04:25:24.110 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T04:25:24.110 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T04:25:24.110 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T04:25:24.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T04:25:24.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T04:25:24.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T04:25:24.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T04:25:24.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T04:25:24.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T04:25:24.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T04:25:24.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T04:25:24.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T04:25:24.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T04:25:24.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T04:25:24.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T04:25:24.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T04:25:24.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T04:25:24.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T04:25:24.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T04:25:24.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T04:25:24.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T04:25:24.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T04:25:24.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T04:25:24.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T04:25:24.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T04:25:24.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T04:25:24.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T04:25:24.115 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T04:25:24.115 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T04:25:24.115 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:24.115 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:25:24.115 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:24.116 INFO:teuthology.orchestra.run.smithi190.stdout:Install 32 Packages 2023-11-15T04:25:24.116 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:24.116 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 31 M 2023-11-15T04:25:24.117 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 111 M 2023-11-15T04:25:24.117 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:25:24.390 INFO:teuthology.orchestra.run.smithi190.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 126 kB/s | 6.3 kB 00:00 2023-11-15T04:25:24.441 INFO:teuthology.orchestra.run.smithi190.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-11-15T04:25:24.500 INFO:teuthology.orchestra.run.smithi190.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-11-15T04:25:24.575 INFO:teuthology.orchestra.run.smithi190.stdout:(4/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 27 MB/s | 5.0 MB 00:00 2023-11-15T04:25:24.600 INFO:teuthology.orchestra.run.smithi190.stdout:(5/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 2.2 MB/s | 220 kB 00:00 2023-11-15T04:25:24.834 INFO:teuthology.orchestra.run.smithi190.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 46 MB/s | 18 MB 00:00 2023-11-15T04:25:24.859 INFO:teuthology.orchestra.run.smithi190.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 150 kB/s | 39 kB 00:00 2023-11-15T04:25:24.887 INFO:teuthology.orchestra.run.smithi190.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 507 kB/s | 157 kB 00:00 2023-11-15T04:25:24.977 INFO:teuthology.orchestra.run.smithi190.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 488 kB/s | 57 kB 00:00 2023-11-15T04:25:25.068 INFO:teuthology.orchestra.run.smithi190.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.4 MB/s | 457 kB 00:00 2023-11-15T04:25:25.102 INFO:teuthology.orchestra.run.smithi190.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.9 MB/s | 249 kB 00:00 2023-11-15T04:25:25.127 INFO:teuthology.orchestra.run.smithi190.stdout:(12/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 353 kB/s | 103 kB 00:00 2023-11-15T04:25:25.153 INFO:teuthology.orchestra.run.smithi190.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2023-11-15T04:25:25.236 INFO:teuthology.orchestra.run.smithi190.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 3.7 MB/s | 641 kB 00:00 2023-11-15T04:25:25.262 INFO:teuthology.orchestra.run.smithi190.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 326 kB/s | 44 kB 00:00 2023-11-15T04:25:25.314 INFO:teuthology.orchestra.run.smithi190.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2023-11-15T04:25:25.354 INFO:teuthology.orchestra.run.smithi190.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 283 kB/s | 11 kB 00:00 2023-11-15T04:25:25.379 INFO:teuthology.orchestra.run.smithi190.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.3 MB/s | 384 kB 00:00 2023-11-15T04:25:25.405 INFO:teuthology.orchestra.run.smithi190.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 350 kB/s | 18 kB 00:00 2023-11-15T04:25:25.430 INFO:teuthology.orchestra.run.smithi190.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 894 kB/s | 173 kB 00:00 2023-11-15T04:25:25.455 INFO:teuthology.orchestra.run.smithi190.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 647 kB/s | 49 kB 00:00 2023-11-15T04:25:25.480 INFO:teuthology.orchestra.run.smithi190.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 780 kB/s | 59 kB 00:00 2023-11-15T04:25:25.506 INFO:teuthology.orchestra.run.smithi190.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-11-15T04:25:25.531 INFO:teuthology.orchestra.run.smithi190.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2023-11-15T04:25:25.556 INFO:teuthology.orchestra.run.smithi190.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 477 kB/s | 24 kB 00:00 2023-11-15T04:25:25.581 INFO:teuthology.orchestra.run.smithi190.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 583 kB/s | 29 kB 00:00 2023-11-15T04:25:25.607 INFO:teuthology.orchestra.run.smithi190.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 550 kB/s | 27 kB 00:00 2023-11-15T04:25:25.632 INFO:teuthology.orchestra.run.smithi190.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-11-15T04:25:25.658 INFO:teuthology.orchestra.run.smithi190.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-11-15T04:25:25.682 INFO:teuthology.orchestra.run.smithi190.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-11-15T04:25:25.708 INFO:teuthology.orchestra.run.smithi190.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 459 kB/s | 23 kB 00:00 2023-11-15T04:25:25.733 INFO:teuthology.orchestra.run.smithi190.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.0 MB/s | 251 kB 00:00 2023-11-15T04:25:25.737 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:25:25.737 INFO:teuthology.orchestra.run.smithi190.stdout:Total 19 MB/s | 31 MB 00:01 2023-11-15T04:25:25.820 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:25:25.859 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:25:25.859 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:25:26.188 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:25:26.191 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:25:27.309 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:25:27.363 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:25:27.507 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T04:25:27.716 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:25:27.718 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T04:25:27.993 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T04:25:28.027 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:25:28.027 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T04:25:28.028 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T04:25:28.028 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T04:25:28.028 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T04:25:28.029 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T04:25:28.200 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T04:25:28.382 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T04:25:28.603 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T04:25:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T04:25:28.798 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T04:25:28.799 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:28.799 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:25:28.799 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T04:25:28.799 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T04:25:28.800 INFO:teuthology.orchestra.run.smithi047.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T04:25:28.800 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:28.800 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:25:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T04:25:29.053 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph 2023-11-15T04:25:29.168 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T04:25:29.397 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T04:25:29.615 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T04:25:29.907 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T04:25:30.037 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:00 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:25:30.262 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T04:25:30.538 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T04:25:30.710 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T04:25:30.959 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T04:25:31.169 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:25:31.461 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T04:25:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T04:25:31.862 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T04:25:31.893 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:25:31.897 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:31.897 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2023-11-15T04:25:31.898 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:31.898 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:25:31.898 INFO:teuthology.orchestra.run.smithi047.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T04:25:31.898 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:25:31.898 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T04:25:31.899 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T04:25:31.899 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T04:25:31.899 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T04:25:31.899 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T04:25:31.899 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T04:25:31.900 INFO:teuthology.orchestra.run.smithi047.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T04:25:31.900 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T04:25:31.900 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T04:25:31.900 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T04:25:31.900 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T04:25:31.901 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T04:25:31.901 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T04:25:31.901 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T04:25:31.901 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T04:25:31.901 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T04:25:31.902 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T04:25:31.902 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T04:25:31.902 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T04:25:31.902 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T04:25:31.902 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T04:25:31.903 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T04:25:31.903 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T04:25:31.903 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T04:25:31.903 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T04:25:31.903 INFO:teuthology.orchestra.run.smithi047.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T04:25:31.904 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T04:25:31.904 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T04:25:31.904 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T04:25:31.904 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T04:25:31.904 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T04:25:31.905 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:31.905 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:25:31.905 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:31.905 INFO:teuthology.orchestra.run.smithi047.stdout:Install 32 Packages 2023-11-15T04:25:31.905 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:31.906 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 31 M 2023-11-15T04:25:31.906 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 111 M 2023-11-15T04:25:31.907 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:25:32.062 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T04:25:32.260 INFO:teuthology.orchestra.run.smithi047.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 106 kB/s | 6.3 kB 00:00 2023-11-15T04:25:32.298 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T04:25:32.318 INFO:teuthology.orchestra.run.smithi047.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2023-11-15T04:25:32.377 INFO:teuthology.orchestra.run.smithi047.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2023-11-15T04:25:32.410 INFO:teuthology.orchestra.run.smithi047.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-11-15T04:25:32.495 INFO:teuthology.orchestra.run.smithi047.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2023-11-15T04:25:32.554 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T04:25:32.753 INFO:teuthology.orchestra.run.smithi047.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 42 MB/s | 18 MB 00:00 2023-11-15T04:25:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T04:25:32.778 INFO:teuthology.orchestra.run.smithi047.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 137 kB/s | 39 kB 00:00 2023-11-15T04:25:32.804 INFO:teuthology.orchestra.run.smithi047.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 400 kB/s | 157 kB 00:00 2023-11-15T04:25:32.921 INFO:teuthology.orchestra.run.smithi047.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 402 kB/s | 57 kB 00:00 2023-11-15T04:25:33.013 INFO:teuthology.orchestra.run.smithi047.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.1 MB/s | 457 kB 00:00 2023-11-15T04:25:33.055 INFO:teuthology.orchestra.run.smithi047.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.8 MB/s | 249 kB 00:00 2023-11-15T04:25:33.070 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T04:25:33.080 INFO:teuthology.orchestra.run.smithi047.stdout:(12/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 317 kB/s | 103 kB 00:00 2023-11-15T04:25:33.105 INFO:teuthology.orchestra.run.smithi047.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2023-11-15T04:25:33.205 INFO:teuthology.orchestra.run.smithi047.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 3.3 MB/s | 641 kB 00:00 2023-11-15T04:25:33.339 INFO:teuthology.orchestra.run.smithi047.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 168 kB/s | 44 kB 00:00 2023-11-15T04:25:33.356 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T04:25:33.414 INFO:teuthology.orchestra.run.smithi047.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 600 kB/s | 185 kB 00:00 2023-11-15T04:25:33.473 INFO:teuthology.orchestra.run.smithi047.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 191 kB/s | 11 kB 00:00 2023-11-15T04:25:33.515 INFO:teuthology.orchestra.run.smithi047.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2023-11-15T04:25:33.540 INFO:teuthology.orchestra.run.smithi047.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 516 kB/s | 173 kB 00:00 2023-11-15T04:25:33.557 INFO:teuthology.orchestra.run.smithi047.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2023-11-15T04:25:33.591 INFO:teuthology.orchestra.run.smithi047.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 645 kB/s | 49 kB 00:00 2023-11-15T04:25:33.666 INFO:teuthology.orchestra.run.smithi047.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2023-11-15T04:25:33.725 INFO:teuthology.orchestra.run.smithi047.stdout:(23/32): python3-more-itertools-7.2.0-3.el8.noa 319 kB/s | 59 kB 00:00 2023-11-15T04:25:33.733 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T04:25:33.750 INFO:teuthology.orchestra.run.smithi047.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-11-15T04:25:33.775 INFO:teuthology.orchestra.run.smithi047.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2023-11-15T04:25:33.840 INFO:teuthology.orchestra.run.smithi047.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 208 kB/s | 24 kB 00:00 2023-11-15T04:25:33.867 INFO:teuthology.orchestra.run.smithi047.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 318 kB/s | 29 kB 00:00 2023-11-15T04:25:33.893 INFO:teuthology.orchestra.run.smithi047.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2023-11-15T04:25:33.920 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:25:33.985 INFO:teuthology.orchestra.run.smithi047.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 935 kB/s | 85 kB 00:00 2023-11-15T04:25:34.018 INFO:teuthology.orchestra.run.smithi047.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.0 MB/s | 251 kB 00:00 2023-11-15T04:25:34.060 INFO:teuthology.orchestra.run.smithi047.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2023-11-15T04:25:34.160 INFO:teuthology.orchestra.run.smithi047.stdout:(32/32): python3-waitress-1.4.3-1.el8.noarch.rp 799 kB/s | 255 kB 00:00 2023-11-15T04:25:34.162 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:25:34.162 INFO:teuthology.orchestra.run.smithi047.stdout:Total 14 MB/s | 31 MB 00:02 2023-11-15T04:25:34.251 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:25:34.294 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:25:34.295 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:25:34.646 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:25:34.649 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:25:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:25:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:25:35.783 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:25:35.999 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T04:25:36.127 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:25:36.209 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T04:25:36.378 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:25:36.496 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T04:25:36.601 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:25:36.715 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T04:25:36.793 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:25:36.811 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:25:36.904 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T04:25:36.967 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:25:37.095 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T04:25:37.311 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T04:25:37.372 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:25:37.373 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T04:25:37.373 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T04:25:37.373 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T04:25:37.373 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T04:25:37.374 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T04:25:37.374 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T04:25:37.374 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T04:25:37.374 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T04:25:37.375 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T04:25:37.375 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T04:25:37.375 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T04:25:37.375 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T04:25:37.375 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T04:25:37.376 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T04:25:37.376 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T04:25:37.376 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T04:25:37.376 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:25:37.377 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T04:25:37.377 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T04:25:37.377 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T04:25:37.377 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T04:25:37.377 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T04:25:37.378 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T04:25:37.378 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T04:25:37.378 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T04:25:37.378 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T04:25:37.378 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T04:25:37.379 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T04:25:37.379 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T04:25:37.379 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T04:25:37.380 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T04:25:37.480 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T04:25:37.710 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T04:25:38.071 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T04:25:38.320 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T04:25:38.455 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T04:25:38.455 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:38.455 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:25:38.455 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:38.456 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:38.456 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:38.456 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T04:25:38.456 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:38.457 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:38.457 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T04:25:38.457 INFO:teuthology.orchestra.run.smithi190.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T04:25:38.457 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T04:25:38.457 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T04:25:38.458 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T04:25:38.458 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T04:25:38.458 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T04:25:38.458 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T04:25:38.458 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T04:25:38.459 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T04:25:38.459 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T04:25:38.459 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T04:25:38.460 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T04:25:38.460 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T04:25:38.460 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T04:25:38.460 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T04:25:38.460 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T04:25:38.461 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T04:25:38.461 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T04:25:38.461 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T04:25:38.461 INFO:teuthology.orchestra.run.smithi190.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T04:25:38.461 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T04:25:38.461 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T04:25:38.462 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T04:25:38.462 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T04:25:38.462 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T04:25:38.462 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:38.463 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:25:38.543 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T04:25:38.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2023-11-15T04:25:38.933 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T04:25:39.198 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T04:25:39.384 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T04:25:39.505 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:12 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:25:39.658 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T04:25:39.860 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:25:40.206 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T04:25:40.424 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T04:25:40.762 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T04:25:40.965 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:25:40.974 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T04:25:41.183 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:25:41.184 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2023-11-15T04:25:41.184 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:25:41.254 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T04:25:41.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2023-11-15T04:25:41.543 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T04:25:41.796 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T04:25:42.139 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:15 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:25:42.169 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T04:25:42.496 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T04:25:42.884 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T04:25:43.114 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:25:43.806 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:25:43.807 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:43.807 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:25:43.807 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:43.807 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:25:43.808 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T04:25:43.808 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:43.808 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:25:43.809 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:43.809 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2023-11-15T04:25:43.809 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:43.810 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 71 k 2023-11-15T04:25:43.810 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 290 k 2023-11-15T04:25:43.810 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:25:43.848 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 71 kB 00:00 2023-11-15T04:25:43.849 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:25:43.849 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.6 MB/s | 71 kB 00:00 2023-11-15T04:25:43.850 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:25:43.858 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:25:43.858 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:25:43.858 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:25:43.861 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:25:43.903 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:25:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:25:44.615 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:25:44.815 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:25:44.849 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:25:44.988 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:25:45.470 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:25:45.529 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:25:45.529 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:45.530 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:25:45.530 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T04:25:45.530 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:45.531 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:25:45.705 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:25:45.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T04:25:45.953 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:25:46.132 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:25:46.154 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:25:46.289 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:25:46.601 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:25:47.866 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:25:47.866 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T04:25:47.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T04:25:47.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T04:25:47.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T04:25:47.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T04:25:47.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T04:25:47.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T04:25:47.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T04:25:47.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T04:25:47.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T04:25:47.871 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T04:25:47.872 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T04:25:47.872 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T04:25:47.872 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T04:25:47.872 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T04:25:47.872 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T04:25:47.873 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:25:47.873 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T04:25:47.873 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T04:25:47.873 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T04:25:47.874 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T04:25:47.874 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T04:25:47.874 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T04:25:47.875 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T04:25:47.875 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T04:25:47.875 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T04:25:47.876 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T04:25:47.876 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T04:25:47.876 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T04:25:47.876 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T04:25:47.876 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T04:25:48.277 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:25:48.277 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:48.278 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2023-11-15T04:25:48.278 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:48.278 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:25:48.279 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T04:25:48.279 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:48.282 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:25:48.283 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2023-11-15T04:25:48.284 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:48.285 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 163 k 2023-11-15T04:25:48.285 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 587 k 2023-11-15T04:25:48.286 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:25:48.314 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.8 MB/s | 163 kB 00:00 2023-11-15T04:25:48.320 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:25:48.321 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.7 MB/s | 163 kB 00:00 2023-11-15T04:25:48.321 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:25:48.326 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:25:48.326 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:25:48.373 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:25:48.376 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:25:48.428 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T04:25:48.434 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:48.435 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:25:48.436 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:48.436 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:48.437 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:48.437 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T04:25:48.438 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:48.439 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:48.439 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T04:25:48.439 INFO:teuthology.orchestra.run.smithi047.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T04:25:48.440 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T04:25:48.440 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T04:25:48.441 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T04:25:48.441 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T04:25:48.442 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T04:25:48.442 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T04:25:48.443 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T04:25:48.443 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T04:25:48.443 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T04:25:48.444 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T04:25:48.444 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T04:25:48.444 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T04:25:48.444 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T04:25:48.444 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T04:25:48.445 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T04:25:48.445 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T04:25:48.445 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T04:25:48.445 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T04:25:48.446 INFO:teuthology.orchestra.run.smithi047.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T04:25:48.446 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T04:25:48.446 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T04:25:48.449 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T04:25:48.449 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T04:25:48.450 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T04:25:48.450 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:48.450 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:25:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:25:48.638 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base 2023-11-15T04:25:48.716 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:25:49.137 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:25:49.634 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:25:49.803 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:25:49.803 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:49.803 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:25:49.804 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T04:25:49.804 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:49.804 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:25:50.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2023-11-15T04:25:50.904 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:23 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:25:51.237 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:25:51.480 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:25:51.481 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2023-11-15T04:25:51.481 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:25:51.581 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install cephadm 2023-11-15T04:25:52.364 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:25:52.577 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:22 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:25:52.583 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:25:52.585 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2023-11-15T04:25:52.585 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:25:53.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T04:25:54.031 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:25:54.445 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:25:54.445 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:54.446 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:25:54.446 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:54.446 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:25:54.446 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T04:25:54.447 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:54.447 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:25:54.447 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:54.447 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2023-11-15T04:25:54.447 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:54.448 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 71 k 2023-11-15T04:25:54.448 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 290 k 2023-11-15T04:25:54.448 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:25:54.483 INFO:teuthology.orchestra.run.smithi047.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 71 kB 00:00 2023-11-15T04:25:54.484 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:25:54.484 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.9 MB/s | 71 kB 00:00 2023-11-15T04:25:54.484 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:25:54.492 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:25:54.492 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:25:54.494 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:25:54.498 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:25:54.546 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:25:55.050 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:25:55.250 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:25:55.513 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:25:55.701 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:25:55.702 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:55.702 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-11-15T04:25:55.702 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:55.703 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:25:55.703 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T04:25:55.703 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:25:55.703 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T04:25:55.703 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T04:25:55.704 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T04:25:55.704 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T04:25:55.705 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T04:25:55.705 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:55.705 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:25:55.706 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:25:55.706 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2023-11-15T04:25:55.707 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:55.708 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.0 M 2023-11-15T04:25:55.708 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 15 M 2023-11-15T04:25:55.709 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:25:55.825 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 462 kB/s | 24 kB 00:00 2023-11-15T04:25:55.850 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 116 kB/s | 8.9 kB 00:00 2023-11-15T04:25:55.909 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2023-11-15T04:25:55.985 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 268 kB/s | 43 kB 00:00 2023-11-15T04:25:56.010 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 211 kB/s | 34 kB 00:00 2023-11-15T04:25:56.160 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 785 kB/s | 196 kB 00:00 2023-11-15T04:25:56.162 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:25:56.162 INFO:teuthology.orchestra.run.smithi190.stdout:Total 6.4 MB/s | 3.0 MB 00:00 2023-11-15T04:25:56.172 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:25:56.181 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:25:56.181 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:25:56.237 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:25:56.238 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:56.238 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:25:56.238 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T04:25:56.238 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:56.238 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:25:56.340 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:25:56.343 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:25:56.462 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T04:25:56.626 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:25:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T04:25:57.082 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T04:25:57.283 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T04:25:57.436 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T04:25:57.458 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:25:58.450 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T04:25:58.849 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:25:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:25:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:25:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T04:25:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T04:25:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T04:25:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T04:25:59.309 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:25:59.310 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:59.310 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2023-11-15T04:25:59.311 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:59.311 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:25:59.311 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T04:25:59.311 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:59.311 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:25:59.312 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:25:59.312 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2023-11-15T04:25:59.312 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:25:59.313 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 163 k 2023-11-15T04:25:59.313 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 587 k 2023-11-15T04:25:59.313 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:25:59.355 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.7 MB/s | 163 kB 00:00 2023-11-15T04:25:59.355 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:25:59.356 INFO:teuthology.orchestra.run.smithi047.stdout:Total 3.6 MB/s | 163 kB 00:00 2023-11-15T04:25:59.356 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:25:59.369 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:25:59.369 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:25:59.422 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:25:59.426 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:25:59.542 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:25:59.754 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:25:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T04:25:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:59.835 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:25:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T04:25:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T04:25:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T04:25:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T04:25:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T04:25:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T04:25:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:25:59.837 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:25:59.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T04:26:00.227 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:00.798 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:33 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:00.978 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:00.978 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:00.978 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:26:00.979 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T04:26:00.979 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:00.979 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:26:01.208 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr 2023-11-15T04:26:02.206 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:32 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:26:02.454 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:02.455 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================================ 2023-11-15T04:26:02.456 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-11-15T04:26:02.456 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================================ 2023-11-15T04:26:02.456 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:26:02.456 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T04:26:02.456 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:26:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T04:26:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T04:26:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T04:26:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T04:26:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T04:26:02.458 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T04:26:02.458 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T04:26:02.458 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:02.458 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:26:02.458 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================================ 2023-11-15T04:26:02.459 INFO:teuthology.orchestra.run.smithi190.stdout:Install 8 Packages 2023-11-15T04:26:02.459 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:02.460 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 37 M 2023-11-15T04:26:02.460 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 210 M 2023-11-15T04:26:02.460 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:26:03.198 INFO:teuthology.orchestra.run.smithi190.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 53 MB/s | 8.6 MB 00:00 2023-11-15T04:26:03.822 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:26:03.968 INFO:teuthology.orchestra.run.smithi190.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.2 MB/s | 4.9 MB 00:00 2023-11-15T04:26:04.064 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:26:04.066 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2023-11-15T04:26:04.066 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:26:04.068 INFO:teuthology.orchestra.run.smithi190.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 4.6 MB/s | 4.7 MB 00:01 2023-11-15T04:26:04.093 INFO:teuthology.orchestra.run.smithi190.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.8 MB/s | 225 kB 00:00 2023-11-15T04:26:04.166 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T04:26:04.169 INFO:teuthology.orchestra.run.smithi190.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.8 MB/s | 3.7 MB 00:00 2023-11-15T04:26:04.278 INFO:teuthology.orchestra.run.smithi190.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2023-11-15T04:26:04.311 INFO:teuthology.orchestra.run.smithi190.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.9 MB/s | 645 kB 00:00 2023-11-15T04:26:04.603 INFO:teuthology.orchestra.run.smithi190.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 26 MB/s | 14 MB 00:00 2023-11-15T04:26:04.604 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:04.604 INFO:teuthology.orchestra.run.smithi190.stdout:Total 17 MB/s | 37 MB 00:02 2023-11-15T04:26:04.834 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:26:04.852 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:26:04.852 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:26:05.147 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:35 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:26:05.180 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:26:05.184 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:26:05.804 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:26:05.988 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:26:06.119 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:26:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:26:06.858 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:26:07.014 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:26:07.015 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:07.015 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2023-11-15T04:26:07.015 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:07.016 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:26:07.016 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T04:26:07.016 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:26:07.016 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T04:26:07.016 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T04:26:07.017 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T04:26:07.017 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T04:26:07.017 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T04:26:07.017 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:07.017 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:26:07.017 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:07.018 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2023-11-15T04:26:07.018 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:07.019 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 3.0 M 2023-11-15T04:26:07.019 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 15 M 2023-11-15T04:26:07.019 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:26:07.030 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:26:07.645 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:26:07.839 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:26:08.416 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 501 kB/s | 24 kB 00:00 2023-11-15T04:26:08.441 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 122 kB/s | 8.9 kB 00:00 2023-11-15T04:26:08.500 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2023-11-15T04:26:08.532 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:26:08.584 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 254 kB/s | 43 kB 00:00 2023-11-15T04:26:08.832 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 86 kB/s | 34 kB 00:00 2023-11-15T04:26:09.052 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 356 kB/s | 196 kB 00:00 2023-11-15T04:26:09.056 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:09.056 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.4 MB/s | 3.0 MB 00:02 2023-11-15T04:26:09.069 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:26:09.080 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:26:09.080 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:26:09.270 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:26:09.274 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:26:09.338 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:26:09.561 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:26:09.841 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T04:26:10.103 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:26:10.108 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T04:26:10.322 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T04:26:10.505 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T04:26:11.617 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T04:26:11.882 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:26:12.001 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:26:12.335 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:26:12.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:26:12.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T04:26:12.336 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T04:26:12.336 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T04:26:12.336 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T04:26:13.081 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T04:26:13.081 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:13.081 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:26:13.081 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T04:26:13.082 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T04:26:13.082 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T04:26:13.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T04:26:13.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T04:26:13.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T04:26:13.083 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:13.083 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:26:13.177 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T04:26:13.200 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T04:26:13.353 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:26:13.655 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:26:13.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T04:26:13.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T04:26:13.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:26:13.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T04:26:13.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:26:13.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T04:26:13.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T04:26:14.196 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:44 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:26:14.363 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T04:26:14.363 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:14.363 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:26:14.364 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T04:26:14.364 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T04:26:14.364 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T04:26:14.364 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T04:26:14.365 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T04:26:14.365 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T04:26:14.365 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T04:26:14.365 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T04:26:14.365 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:14.366 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:14.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2023-11-15T04:26:15.366 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:48 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:16.043 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:26:16.044 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================================ 2023-11-15T04:26:16.044 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2023-11-15T04:26:16.044 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================================ 2023-11-15T04:26:16.045 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:26:16.045 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T04:26:16.045 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:26:16.045 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T04:26:16.045 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T04:26:16.046 INFO:teuthology.orchestra.run.smithi047.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T04:26:16.046 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T04:26:16.046 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T04:26:16.046 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T04:26:16.046 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T04:26:16.047 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:16.047 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:26:16.047 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================================ 2023-11-15T04:26:16.047 INFO:teuthology.orchestra.run.smithi047.stdout:Install 8 Packages 2023-11-15T04:26:16.047 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:16.048 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 37 M 2023-11-15T04:26:16.048 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 210 M 2023-11-15T04:26:16.048 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:26:16.300 INFO:teuthology.orchestra.run.smithi047.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 49 MB/s | 8.6 MB 00:00 2023-11-15T04:26:16.692 INFO:teuthology.orchestra.run.smithi047.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.6 MB/s | 4.9 MB 00:00 2023-11-15T04:26:16.759 INFO:teuthology.orchestra.run.smithi047.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 7.5 MB/s | 4.7 MB 00:00 2023-11-15T04:26:16.784 INFO:teuthology.orchestra.run.smithi047.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-11-15T04:26:16.926 INFO:teuthology.orchestra.run.smithi047.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 4.5 MB/s | 645 kB 00:00 2023-11-15T04:26:16.984 INFO:teuthology.orchestra.run.smithi047.stdout:(6/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.5 MB/s | 3.7 MB 00:00 2023-11-15T04:26:17.009 INFO:teuthology.orchestra.run.smithi047.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 172 kB 00:00 2023-11-15T04:26:17.099 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:17.101 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:17.101 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-11-15T04:26:17.101 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:17.101 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:26:17.101 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T04:26:17.102 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:26:17.102 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T04:26:17.102 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T04:26:17.102 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T04:26:17.103 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T04:26:17.103 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T04:26:17.103 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T04:26:17.103 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T04:26:17.103 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T04:26:17.105 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T04:26:17.105 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T04:26:17.105 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T04:26:17.105 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T04:26:17.106 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:17.106 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:26:17.106 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:17.106 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2023-11-15T04:26:17.106 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:17.107 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 1.9 M 2023-11-15T04:26:17.107 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 24 M 2023-11-15T04:26:17.108 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:26:17.293 INFO:teuthology.orchestra.run.smithi047.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 26 MB/s | 14 MB 00:00 2023-11-15T04:26:17.294 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:17.294 INFO:teuthology.orchestra.run.smithi047.stdout:Total 29 MB/s | 37 MB 00:01 2023-11-15T04:26:17.536 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 687 kB/s | 27 kB 00:00 2023-11-15T04:26:17.545 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:26:17.565 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:26:17.566 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:26:17.695 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 101 kB/s | 20 kB 00:00 2023-11-15T04:26:17.962 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:26:17.962 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:26:17.963 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 119 kB/s | 27 kB 00:00 2023-11-15T04:26:17.963 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 445 kB/s | 126 kB 00:00 2023-11-15T04:26:17.963 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.2 MB/s | 155 kB 00:00 2023-11-15T04:26:17.964 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 625 kB/s | 110 kB 00:00 2023-11-15T04:26:17.964 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 391 kB/s | 43 kB 00:00 2023-11-15T04:26:17.996 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 219 kB/s | 33 kB 00:00 2023-11-15T04:26:18.021 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 108 kB/s | 16 kB 00:00 2023-11-15T04:26:18.080 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 543 kB/s | 81 kB 00:00 2023-11-15T04:26:18.105 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 769 kB/s | 64 kB 00:00 2023-11-15T04:26:18.130 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-11-15T04:26:18.164 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 7.0 MB/s | 1.2 MB 00:00 2023-11-15T04:26:18.167 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:18.167 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.8 MB/s | 1.9 MB 00:01 2023-11-15T04:26:18.225 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:26:18.241 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:26:18.241 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:26:18.328 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:26:18.331 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:26:18.575 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:26:18.825 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:26:18.885 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:26:19.018 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:26:19.073 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T04:26:19.177 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:26:19.261 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T04:26:19.496 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T04:26:19.734 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T04:26:19.836 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:26:19.941 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T04:26:20.093 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:26:20.190 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T04:26:20.493 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T04:26:20.732 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T04:26:20.771 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:26:20.943 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:26:21.716 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:26:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T04:26:22.059 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T04:26:22.068 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:26:22.284 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T04:26:22.326 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:26:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T04:26:22.627 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:26:22.931 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:26:22.931 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T04:26:22.932 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T04:26:22.932 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T04:26:22.932 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T04:26:22.932 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T04:26:22.932 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T04:26:22.933 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T04:26:22.933 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T04:26:22.933 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T04:26:22.933 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T04:26:22.933 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T04:26:22.934 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T04:26:23.662 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T04:26:23.663 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:23.663 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:26:23.663 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T04:26:23.663 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T04:26:23.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T04:26:23.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T04:26:23.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T04:26:23.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T04:26:23.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T04:26:23.665 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T04:26:23.665 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T04:26:23.665 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T04:26:23.665 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T04:26:23.665 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T04:26:23.666 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T04:26:23.666 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:23.666 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:23.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T04:26:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:26:24.638 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:57 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:25.699 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T04:26:25.867 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:26:26.240 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:26:26.240 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T04:26:26.241 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T04:26:26.241 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:26:26.241 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T04:26:26.241 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:26:26.242 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T04:26:26.242 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T04:26:26.298 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:26.299 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:26.299 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-11-15T04:26:26.299 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:26.299 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:26:26.300 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T04:26:26.300 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:26:26.300 INFO:teuthology.orchestra.run.smithi190.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T04:26:26.300 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T04:26:26.300 INFO:teuthology.orchestra.run.smithi190.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T04:26:26.301 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T04:26:26.301 INFO:teuthology.orchestra.run.smithi190.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T04:26:26.301 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:26.301 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:26:26.301 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:26.302 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2023-11-15T04:26:26.302 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:26.302 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 5.8 M 2023-11-15T04:26:26.303 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 24 M 2023-11-15T04:26:26.303 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:26:26.580 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 89 kB 00:00 2023-11-15T04:26:26.705 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 169 kB/s | 21 kB 00:00 2023-11-15T04:26:26.881 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.0 MB/s | 355 kB 00:00 2023-11-15T04:26:26.906 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.4 MB/s | 538 kB 00:00 2023-11-15T04:26:26.940 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 837 kB/s | 49 kB 00:00 2023-11-15T04:26:27.040 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T04:26:27.040 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:27.040 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:26:27.040 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T04:26:27.041 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T04:26:27.041 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T04:26:27.041 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T04:26:27.041 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T04:26:27.041 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T04:26:27.042 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T04:26:27.042 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T04:26:27.042 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:27.042 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:26:27.157 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.6 MB/s | 4.8 MB 00:00 2023-11-15T04:26:27.157 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:27.158 INFO:teuthology.orchestra.run.smithi190.stdout:Total 6.8 MB/s | 5.8 MB 00:00 2023-11-15T04:26:27.170 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-rook 2023-11-15T04:26:27.207 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:26:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:26:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:26:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:26:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:26:27.597 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:26:27.834 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T04:26:28.153 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T04:26:28.172 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:58 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:26:29.086 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T04:26:29.453 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T04:26:29.671 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T04:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:26:30.020 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:26:30.021 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:30.022 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2023-11-15T04:26:30.022 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:30.022 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:26:30.022 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T04:26:30.022 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:26:30.023 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T04:26:30.023 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T04:26:30.023 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T04:26:30.023 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T04:26:30.023 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T04:26:30.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T04:26:30.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T04:26:30.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T04:26:30.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T04:26:30.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T04:26:30.025 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T04:26:30.025 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T04:26:30.025 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:30.025 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:26:30.026 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:30.026 INFO:teuthology.orchestra.run.smithi047.stdout:Install 13 Packages 2023-11-15T04:26:30.026 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:30.027 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 1.9 M 2023-11-15T04:26:30.027 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 24 M 2023-11-15T04:26:30.027 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:26:30.138 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:26:30.139 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:26:30.139 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T04:26:30.139 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T04:26:30.139 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T04:26:30.139 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T04:26:30.201 INFO:teuthology.orchestra.run.smithi047.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 721 kB/s | 27 kB 00:00 2023-11-15T04:26:30.343 INFO:teuthology.orchestra.run.smithi047.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 148 kB/s | 27 kB 00:00 2023-11-15T04:26:30.368 INFO:teuthology.orchestra.run.smithi047.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 98 kB/s | 20 kB 00:00 2023-11-15T04:26:30.485 INFO:teuthology.orchestra.run.smithi047.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.3 MB/s | 155 kB 00:00 2023-11-15T04:26:30.510 INFO:teuthology.orchestra.run.smithi047.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 409 kB/s | 126 kB 00:00 2023-11-15T04:26:30.536 INFO:teuthology.orchestra.run.smithi047.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 572 kB/s | 110 kB 00:00 2023-11-15T04:26:30.561 INFO:teuthology.orchestra.run.smithi047.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 564 kB/s | 43 kB 00:00 2023-11-15T04:26:30.761 INFO:teuthology.orchestra.run.smithi047.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 132 kB/s | 33 kB 00:00 2023-11-15T04:26:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 65 kB/s | 16 kB 00:00 2023-11-15T04:26:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 335 kB/s | 81 kB 00:00 2023-11-15T04:26:30.816 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T04:26:30.817 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:30.817 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:26:30.817 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T04:26:30.817 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T04:26:30.818 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T04:26:30.818 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:30.819 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:30.862 INFO:teuthology.orchestra.run.smithi047.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-11-15T04:26:30.887 INFO:teuthology.orchestra.run.smithi047.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 639 kB/s | 64 kB 00:00 2023-11-15T04:26:30.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2023-11-15T04:26:31.154 INFO:teuthology.orchestra.run.smithi047.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.0 MB/s | 1.2 MB 00:00 2023-11-15T04:26:31.157 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:31.157 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2023-11-15T04:26:31.221 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:26:31.238 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:26:31.239 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:26:31.339 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:26:31.343 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:26:31.805 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:04 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:31.856 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:26:32.094 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T04:26:32.364 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T04:26:32.723 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T04:26:32.991 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T04:26:33.243 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T04:26:33.458 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:33.458 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:33.459 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:26:33.459 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:33.459 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:26:33.459 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T04:26:33.459 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-15T04:26:33.460 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T04:26:33.460 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:33.460 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:26:33.460 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:33.460 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2023-11-15T04:26:33.461 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:33.461 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 968 k 2023-11-15T04:26:33.461 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 3.2 M 2023-11-15T04:26:33.462 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:26:33.505 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T04:26:33.552 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-11-15T04:26:33.627 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 648 kB/s | 83 kB 00:00 2023-11-15T04:26:33.628 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:33.628 INFO:teuthology.orchestra.run.smithi190.stdout:Total 5.6 MB/s | 968 kB 00:00 2023-11-15T04:26:33.639 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:26:33.657 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:26:33.659 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:26:33.742 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:26:33.745 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:26:33.807 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T04:26:33.912 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:26:34.052 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T04:26:34.175 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T04:26:34.392 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:26:34.780 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:26:34.780 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T04:26:35.203 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T04:26:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T04:26:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:35.457 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:26:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T04:26:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:35.458 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:35.541 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T04:26:35.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2023-11-15T04:26:35.783 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T04:26:36.008 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T04:26:36.166 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:26:36.514 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:26:36.515 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T04:26:36.515 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T04:26:36.515 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T04:26:36.515 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T04:26:36.516 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T04:26:36.516 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T04:26:36.516 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T04:26:36.516 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T04:26:36.517 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T04:26:36.517 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T04:26:36.517 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T04:26:36.517 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T04:26:36.537 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:09 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:37.458 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T04:26:37.458 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:37.458 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:26:37.459 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T04:26:37.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T04:26:37.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T04:26:37.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T04:26:37.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T04:26:37.460 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T04:26:37.460 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T04:26:37.460 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T04:26:37.460 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T04:26:37.460 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T04:26:37.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T04:26:37.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T04:26:37.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T04:26:37.461 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:37.461 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:26:37.583 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T04:26:38.194 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:38.195 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:38.195 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:26:38.195 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:38.195 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:26:38.195 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T04:26:38.196 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:38.196 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:26:38.196 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:38.196 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2023-11-15T04:26:38.196 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:38.197 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 131 k 2023-11-15T04:26:38.197 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 439 k 2023-11-15T04:26:38.197 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:26:38.240 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 131 kB 00:00 2023-11-15T04:26:38.241 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:38.241 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.8 MB/s | 131 kB 00:00 2023-11-15T04:26:38.241 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:26:38.246 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:26:38.246 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:26:38.295 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:26:38.298 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:26:38.411 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:26:38.575 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:08 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:26:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:38.927 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:39.568 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:26:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:26:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:39.569 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:39.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2023-11-15T04:26:40.421 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:26:40.422 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:40.423 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2023-11-15T04:26:40.423 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:40.423 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:26:40.423 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T04:26:40.424 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:26:40.424 INFO:teuthology.orchestra.run.smithi047.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T04:26:40.424 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T04:26:40.424 INFO:teuthology.orchestra.run.smithi047.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T04:26:40.424 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T04:26:40.425 INFO:teuthology.orchestra.run.smithi047.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T04:26:40.425 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:40.425 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:26:40.425 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:40.426 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2023-11-15T04:26:40.426 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:40.427 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 5.8 M 2023-11-15T04:26:40.427 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 24 M 2023-11-15T04:26:40.427 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:26:40.640 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:13 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:41.020 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.4 MB/s | 89 kB 00:00 2023-11-15T04:26:41.162 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 149 kB/s | 21 kB 00:00 2023-11-15T04:26:41.404 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.4 MB/s | 355 kB 00:00 2023-11-15T04:26:41.470 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 734 kB/s | 49 kB 00:00 2023-11-15T04:26:41.571 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 917 kB/s | 538 kB 00:00 2023-11-15T04:26:42.098 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:26:42.179 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.0 MB/s | 4.8 MB 00:01 2023-11-15T04:26:42.180 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:42.181 INFO:teuthology.orchestra.run.smithi047.stdout:Total 3.3 MB/s | 5.8 MB 00:01 2023-11-15T04:26:42.237 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:26:42.248 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:26:42.249 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:26:42.315 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:42.316 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2023-11-15T04:26:42.316 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:42.367 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:26:42.370 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:26:42.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2023-11-15T04:26:42.666 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:26:42.960 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T04:26:43.275 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:16 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:43.399 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T04:26:44.450 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T04:26:44.947 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:44.948 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:44.948 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:26:44.949 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:44.949 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:26:44.949 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T04:26:44.949 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:44.949 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:26:44.950 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:26:44.950 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2023-11-15T04:26:44.950 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:44.950 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 24 k 2023-11-15T04:26:44.950 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 94 k 2023-11-15T04:26:44.951 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:26:44.987 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T04:26:45.004 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 439 kB/s | 24 kB 00:00 2023-11-15T04:26:45.005 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:45.005 INFO:teuthology.orchestra.run.smithi190.stdout:Total 430 kB/s | 24 kB 00:00 2023-11-15T04:26:45.006 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:26:45.009 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:26:45.009 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:26:45.011 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:26:45.014 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:26:45.062 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:26:45.224 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T04:26:45.239 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:45.406 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:26:45.507 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:45.726 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:26:45.726 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:26:45.727 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T04:26:45.727 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T04:26:45.727 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T04:26:45.727 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T04:26:46.125 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:46.125 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:46.126 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:26:46.126 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:26:46.126 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:26:46.126 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:46.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2023-11-15T04:26:46.661 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T04:26:46.662 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:46.662 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:26:46.662 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T04:26:46.663 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T04:26:46.663 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T04:26:46.663 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:46.664 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:26:46.787 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-fuse 2023-11-15T04:26:47.101 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:20 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:47.778 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:17 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:26:48.561 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:26:48.775 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:48.776 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2023-11-15T04:26:48.776 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:48.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2023-11-15T04:26:49.646 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:26:49.647 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:49.647 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:26:49.647 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:49.647 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:26:49.647 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T04:26:49.648 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2023-11-15T04:26:49.648 INFO:teuthology.orchestra.run.smithi047.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T04:26:49.648 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:49.648 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:26:49.648 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:49.649 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2023-11-15T04:26:49.649 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:49.649 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 968 k 2023-11-15T04:26:49.650 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 3.2 M 2023-11-15T04:26:49.650 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:26:49.732 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:22 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:49.765 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 11 MB/s | 885 kB 00:00 2023-11-15T04:26:49.832 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 572 kB/s | 83 kB 00:00 2023-11-15T04:26:49.833 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:49.833 INFO:teuthology.orchestra.run.smithi047.stdout:Total 5.1 MB/s | 968 kB 00:00 2023-11-15T04:26:49.846 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:26:49.870 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:26:49.871 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:26:49.975 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:26:49.978 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:26:50.165 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:26:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T04:26:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:26:51.183 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:26:51.183 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T04:26:51.216 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:26:51.430 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:51.431 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2023-11-15T04:26:51.431 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:51.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2023-11-15T04:26:52.165 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T04:26:52.165 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:52.166 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:26:52.166 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T04:26:52.166 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:52.166 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:26:52.386 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:25 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:53.392 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados-devel 2023-11-15T04:26:53.833 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:26:54.052 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:54.053 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2023-11-15T04:26:54.053 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:54.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2023-11-15T04:26:54.403 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:24 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:26:55.034 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:28 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:56.309 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:26:56.310 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:56.310 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:26:56.310 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:56.310 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:26:56.311 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T04:26:56.311 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:56.311 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:26:56.311 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:26:56.311 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2023-11-15T04:26:56.312 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:56.312 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 131 k 2023-11-15T04:26:56.312 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 439 k 2023-11-15T04:26:56.312 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:26:56.356 INFO:teuthology.orchestra.run.smithi047.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 131 kB 00:00 2023-11-15T04:26:56.357 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:26:56.357 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.8 MB/s | 131 kB 00:00 2023-11-15T04:26:56.357 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:26:56.363 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:26:56.363 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:26:56.421 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:26:56.424 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:26:56.489 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:26:56.546 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:26:56.703 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:56.704 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2023-11-15T04:26:56.704 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:56.780 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:56.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2023-11-15T04:26:57.118 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:57.665 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:30 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:26:57.839 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:26:57.839 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:57.840 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:26:57.840 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:26:57.840 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:26:57.840 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:26:58.066 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs2 2023-11-15T04:26:59.071 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:29 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:26:59.092 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:26:59.308 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:26:59.309 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2023-11-15T04:26:59.309 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:26:59.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2023-11-15T04:27:00.259 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:33 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:27:00.690 INFO:teuthology.orchestra.run.smithi047.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:27:00.930 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:00.931 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2023-11-15T04:27:00.931 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:01.036 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs-devel 2023-11-15T04:27:01.700 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:27:01.914 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:27:01.915 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2023-11-15T04:27:01.916 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:27:02.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2023-11-15T04:27:02.033 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:32 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:02.874 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:35 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:27:03.888 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:03.889 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:03.889 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:27:03.889 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:03.890 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:27:03.890 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T04:27:03.890 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:03.890 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:27:03.890 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:03.891 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2023-11-15T04:27:03.891 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:03.891 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 24 k 2023-11-15T04:27:03.891 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 94 k 2023-11-15T04:27:03.892 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:27:03.926 INFO:teuthology.orchestra.run.smithi047.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 663 kB/s | 24 kB 00:00 2023-11-15T04:27:03.927 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:27:03.928 INFO:teuthology.orchestra.run.smithi047.stdout:Total 640 kB/s | 24 kB 00:00 2023-11-15T04:27:03.928 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:27:03.931 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:27:03.932 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:27:03.934 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:27:03.938 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:27:03.990 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:27:04.329 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:04.546 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:27:04.547 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:27:04.548 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:27:04.548 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:27:04.548 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:27:04.548 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T04:27:04.548 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:04.549 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:27:04.549 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:27:04.549 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2023-11-15T04:27:04.549 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:04.549 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 89 k 2023-11-15T04:27:04.550 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 298 k 2023-11-15T04:27:04.550 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:27:04.595 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 89 kB 00:00 2023-11-15T04:27:04.596 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:27:04.596 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.8 MB/s | 89 kB 00:00 2023-11-15T04:27:04.596 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:27:04.605 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:27:04.605 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:27:04.636 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:04.655 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:27:04.658 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:27:04.772 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:27:04.954 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:05.249 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:05.348 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:05.348 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:05.349 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:27:05.349 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:27:05.349 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:05.349 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:05.475 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados2 2023-11-15T04:27:05.950 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:05.951 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:05.951 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:27:05.951 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T04:27:05.951 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:05.952 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:27:06.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2023-11-15T04:27:06.476 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:36 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:07.002 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:40 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:27:08.090 INFO:teuthology.orchestra.run.smithi047.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:27:08.328 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:08.330 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2023-11-15T04:27:08.330 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:08.424 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librbd1 2023-11-15T04:27:08.666 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:27:08.667 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:27:08.667 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:27:08.668 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:27:08.668 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:27:08.668 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:27:08.668 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:08.668 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:27:08.669 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:27:08.669 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2023-11-15T04:27:08.669 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:08.669 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.7 M 2023-11-15T04:27:08.670 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 18 M 2023-11-15T04:27:08.670 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:27:08.765 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 38 MB/s | 3.7 MB 00:00 2023-11-15T04:27:08.766 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:27:08.766 INFO:teuthology.orchestra.run.smithi190.stdout:Total 38 MB/s | 3.7 MB 00:00 2023-11-15T04:27:08.766 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:27:08.778 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:27:08.778 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:27:08.841 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:27:08.844 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:27:09.309 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:27:09.424 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:39 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:09.495 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:09.913 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:10.607 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:10.608 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:10.608 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:27:10.608 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T04:27:10.608 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:10.609 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:27:10.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2023-11-15T04:27:11.046 INFO:teuthology.orchestra.run.smithi047.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:27:11.288 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:11.289 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2023-11-15T04:27:11.289 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:11.382 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rados 2023-11-15T04:27:11.706 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:44 ago on Wed 15 Nov 2023 04:24:27 AM UTC. 2023-11-15T04:27:12.374 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:42 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:13.371 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-15T04:27:13.372 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:27:13.372 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-11-15T04:27:13.372 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:27:13.372 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-15T04:27:13.372 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T04:27:13.373 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:13.373 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-15T04:27:13.373 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-15T04:27:13.373 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2023-11-15T04:27:13.373 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:13.374 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 173 k 2023-11-15T04:27:13.374 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 649 k 2023-11-15T04:27:13.374 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-15T04:27:13.418 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.7 MB/s | 173 kB 00:00 2023-11-15T04:27:13.419 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:27:13.419 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.6 MB/s | 173 kB 00:00 2023-11-15T04:27:13.420 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-15T04:27:13.430 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-15T04:27:13.430 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-15T04:27:13.479 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-15T04:27:13.482 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-15T04:27:13.604 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-15T04:27:13.793 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:13.984 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:27:14.070 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:14.226 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:14.228 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2023-11-15T04:27:14.228 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:14.322 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rgw 2023-11-15T04:27:14.752 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:14.752 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:14.752 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-15T04:27:14.752 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:27:14.753 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:27:14.753 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-15T04:27:14.970 DEBUG:teuthology.parallel:result is None 2023-11-15T04:27:15.314 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:45 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:16.919 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:27:17.160 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:17.162 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2023-11-15T04:27:17.162 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:17.273 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-cephfs 2023-11-15T04:27:18.275 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:48 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:19.904 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:27:20.147 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:20.148 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2023-11-15T04:27:20.148 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:20.252 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rbd 2023-11-15T04:27:21.246 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:51 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:22.845 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:27:23.091 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:23.091 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2023-11-15T04:27:23.092 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:23.188 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-fuse 2023-11-15T04:27:24.187 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:54 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:26.033 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:26.034 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:26.034 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:27:26.034 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:26.034 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:27:26.035 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T04:27:26.035 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:26.035 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:27:26.035 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:26.035 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2023-11-15T04:27:26.036 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:26.036 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 89 k 2023-11-15T04:27:26.036 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 298 k 2023-11-15T04:27:26.036 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:27:26.083 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2023-11-15T04:27:26.084 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:27:26.084 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.8 MB/s | 89 kB 00:00 2023-11-15T04:27:26.084 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:27:26.096 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:27:26.096 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:27:26.152 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:27:26.158 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:27:26.281 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:27:26.535 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:26.862 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:27.615 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:27.615 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:27.615 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:27:27.616 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T04:27:27.616 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:27.616 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:27.837 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-mirror 2023-11-15T04:27:28.831 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:58 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:30.680 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:30.681 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:30.681 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:27:30.682 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:30.682 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:27:30.682 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:27:30.682 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:30.683 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:27:30.683 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:30.683 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2023-11-15T04:27:30.683 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:30.683 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 3.7 M 2023-11-15T04:27:30.684 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 18 M 2023-11-15T04:27:30.684 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:27:30.769 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2023-11-15T04:27:30.770 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:27:30.771 INFO:teuthology.orchestra.run.smithi047.stdout:Total 42 MB/s | 3.7 MB 00:00 2023-11-15T04:27:30.771 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:27:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:27:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:27:30.857 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:27:30.862 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:27:31.369 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:27:31.584 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:32.062 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:32.827 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:32.827 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:32.828 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:27:32.828 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T04:27:32.828 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:32.828 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:33.020 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-nbd 2023-11-15T04:27:34.023 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:03:04 ago on Wed 15 Nov 2023 04:24:30 AM UTC. 2023-11-15T04:27:35.872 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2023-11-15T04:27:35.873 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:35.873 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2023-11-15T04:27:35.873 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:35.873 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2023-11-15T04:27:35.874 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T04:27:35.874 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:35.874 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2023-11-15T04:27:35.874 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2023-11-15T04:27:35.874 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2023-11-15T04:27:35.875 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:35.875 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 173 k 2023-11-15T04:27:35.875 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 649 k 2023-11-15T04:27:35.876 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2023-11-15T04:27:35.916 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.1 MB/s | 173 kB 00:00 2023-11-15T04:27:35.916 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:27:35.917 INFO:teuthology.orchestra.run.smithi047.stdout:Total 4.0 MB/s | 173 kB 00:00 2023-11-15T04:27:35.917 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2023-11-15T04:27:35.929 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2023-11-15T04:27:35.929 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2023-11-15T04:27:35.988 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2023-11-15T04:27:35.991 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2023-11-15T04:27:36.115 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2023-11-15T04:27:36.350 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:36.682 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:37.455 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:27:37.455 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:37.455 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2023-11-15T04:27:37.456 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:27:37.456 INFO:teuthology.orchestra.run.smithi047.stdout: 2023-11-15T04:27:37.456 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2023-11-15T04:27:37.662 DEBUG:teuthology.parallel:result is None 2023-11-15T04:27:37.663 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:27:37.663 INFO:teuthology.packaging:ref: None 2023-11-15T04:27:37.664 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:27:37.664 INFO:teuthology.packaging:branch: None 2023-11-15T04:27:37.664 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:27:37.902 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:27:37.903 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:27:38.047 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T04:27:38.075 INFO:teuthology.orchestra.run.smithi047.stdout:16.2.4-0.el8 2023-11-15T04:27:38.076 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T04:27:38.076 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T04:27:38.077 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:27:38.077 INFO:teuthology.packaging:ref: None 2023-11-15T04:27:38.078 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:27:38.078 INFO:teuthology.packaging:branch: None 2023-11-15T04:27:38.078 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:27:38.272 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:27:38.273 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:27:38.415 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T04:27:38.445 INFO:teuthology.orchestra.run.smithi190.stdout:16.2.4-0.el8 2023-11-15T04:27:38.445 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T04:27:38.446 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T04:27:38.446 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-15T04:27:38.447 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:27:38.447 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T04:27:38.484 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-15T04:27:38.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T04:27:38.516 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-15T04:27:38.517 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:27:38.517 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T04:27:38.558 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T04:27:38.633 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-15T04:27:38.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T04:27:38.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T04:27:38.736 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-15T04:27:38.736 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:27:38.737 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T04:27:38.771 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T04:27:38.846 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-15T04:27:38.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T04:27:38.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T04:27:38.950 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-15T04:27:38.950 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:27:38.951 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T04:27:38.985 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T04:27:39.060 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-15T04:27:39.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T04:27:39.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T04:27:39.161 INFO:teuthology.run_tasks:Running task print... 2023-11-15T04:27:39.171 INFO:teuthology.task.print:**** done install task... 2023-11-15T04:27:39.172 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-15T04:27:39.311 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T04:27:39.311 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-15T04:27:39.312 INFO:tasks.cephadm:Cluster fsid is 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:27:39.313 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-15T04:27:39.313 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-15T04:27:39.313 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi047': '172.21.15.47', 'mon.smithi190': '172.21.15.190'} 2023-11-15T04:27:39.313 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-15T04:27:39.314 DEBUG:teuthology.orchestra.run.smithi047:> sudo hostname $(hostname -s) 2023-11-15T04:27:39.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2023-11-15T04:27:39.384 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-15T04:27:39.384 DEBUG:teuthology.orchestra.run.smithi047:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T04:27:39.591 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 04:27 /home/ubuntu/cephtest/cephadm 2023-11-15T04:27:39.591 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T04:27:39.726 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 04:27 /home/ubuntu/cephtest/cephadm 2023-11-15T04:27:39.727 DEBUG:teuthology.orchestra.run.smithi047:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T04:27:39.751 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T04:27:39.784 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph 2023-11-15T04:27:39.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2023-11-15T04:27:39.858 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /etc/ceph 2023-11-15T04:27:39.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2023-11-15T04:27:39.931 INFO:tasks.cephadm:Writing seed config... 2023-11-15T04:27:39.932 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-15T04:27:39.933 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-15T04:27:39.933 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-15T04:27:39.933 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-15T04:27:39.934 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-15T04:27:39.934 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-15T04:27:39.934 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-15T04:27:39.935 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-15T04:27:39.935 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-15T04:27:39.935 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-15T04:27:39.935 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-15T04:27:39.936 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-15T04:27:39.936 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-15T04:27:39.936 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-15T04:27:39.936 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-15T04:27:39.937 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-15T04:27:39.937 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-15T04:27:39.937 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-15T04:27:39.937 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-15T04:27:39.938 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-15T04:27:39.938 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-15T04:27:39.938 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-15T04:27:39.939 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-15T04:27:39.939 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-15T04:27:39.939 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-15T04:27:39.939 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-15T04:27:39.940 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-15T04:27:39.940 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-15T04:27:39.940 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-15T04:27:39.940 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-15T04:27:39.941 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-15T04:27:39.941 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-15T04:27:39.941 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-15T04:27:39.942 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-15T04:27:39.942 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-15T04:27:39.942 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-15T04:27:39.942 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-15T04:27:39.943 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-15T04:27:39.944 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:27:39.944 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-15T04:27:39.967 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-15T04:27:39.967 DEBUG:teuthology.orchestra.run.smithi047:mon.smithi047> sudo journalctl -f -n 0 -u ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047.service 2023-11-15T04:27:40.010 INFO:tasks.cephadm:Bootstrapping... 2023-11-15T04:27:40.010 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.47 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-15T04:27:40.045 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:-- Logs begin at Wed 2023-11-15 04:15:05 UTC. -- 2023-11-15T04:27:40.310 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T04:27:41.190 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 4.6.1 2023-11-15T04:27:41.200 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying podman|docker is present... 2023-11-15T04:27:41.201 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T04:27:41.225 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 4.6.1 2023-11-15T04:27:41.233 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying lvm2 is present... 2023-11-15T04:27:41.233 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying time synchronization is in place... 2023-11-15T04:27:41.234 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T04:27:41.242 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T04:27:41.242 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active chrony.service 2023-11-15T04:27:41.249 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2023-11-15T04:27:41.249 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T04:27:41.256 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: enabled 2023-11-15T04:27:41.257 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T04:27:41.263 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: active 2023-11-15T04:27:41.264 INFO:teuthology.orchestra.run.smithi047.stderr:Unit chronyd.service is enabled and running 2023-11-15T04:27:41.264 INFO:teuthology.orchestra.run.smithi047.stderr:Repeating the final host check... 2023-11-15T04:27:41.264 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T04:27:41.287 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 4.6.1 2023-11-15T04:27:41.299 INFO:teuthology.orchestra.run.smithi047.stderr:podman|docker (/bin/podman) is present 2023-11-15T04:27:41.299 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl is present 2023-11-15T04:27:41.300 INFO:teuthology.orchestra.run.smithi047.stderr:lvcreate is present 2023-11-15T04:27:41.300 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T04:27:41.306 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T04:27:41.306 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active chrony.service 2023-11-15T04:27:41.312 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2023-11-15T04:27:41.313 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T04:27:41.319 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: enabled 2023-11-15T04:27:41.320 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T04:27:41.325 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: active 2023-11-15T04:27:41.326 INFO:teuthology.orchestra.run.smithi047.stderr:Unit chronyd.service is enabled and running 2023-11-15T04:27:41.326 INFO:teuthology.orchestra.run.smithi047.stderr:Host looks OK 2023-11-15T04:27:41.327 INFO:teuthology.orchestra.run.smithi047.stderr:Cluster fsid: 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:27:41.328 INFO:teuthology.orchestra.run.smithi047.stderr:Acquiring lock 140321730787816 on /run/cephadm/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:27:41.328 INFO:teuthology.orchestra.run.smithi047.stderr:Lock 140321730787816 acquired on /run/cephadm/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:27:41.328 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying IP 172.21.15.47 port 3300 ... 2023-11-15T04:27:41.328 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying IP 172.21.15.47 port 6789 ... 2023-11-15T04:27:41.329 INFO:teuthology.orchestra.run.smithi047.stderr:Base mon IP is 172.21.15.47, final addrv is [v2:172.21.15.47:3300,v1:172.21.15.47:6789] 2023-11-15T04:27:41.329 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /sbin/ip route ls 2023-11-15T04:27:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.47 metric 100 2023-11-15T04:27:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.47 metric 100 2023-11-15T04:27:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /sbin/ip -6 route ls 2023-11-15T04:27:41.334 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-15T04:27:41.334 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-15T04:27:41.334 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1783sec hoplimit 64 pref medium 2023-11-15T04:27:41.335 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /sbin/ip -6 addr ls 2023-11-15T04:27:41.337 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-15T04:27:41.337 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-15T04:27:41.337 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T04:27:41.338 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-11-15T04:27:41.338 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2023-11-15T04:27:41.338 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T04:27:41.339 INFO:teuthology.orchestra.run.smithi047.stderr:Mon IP 172.21.15.47 is in CIDR network 172.21.0.0/20 2023-11-15T04:27:41.339 INFO:teuthology.orchestra.run.smithi047.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-15T04:27:41.340 INFO:teuthology.orchestra.run.smithi047.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-15T04:27:41.340 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-15T04:27:41.439 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-15T04:27:47.902 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Getting image source signatures 2023-11-15T04:27:47.911 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-15T04:27:47.911 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-15T04:28:02.494 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T04:28:02.560 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Writing manifest to image destination 2023-11-15T04:28:02.891 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T04:28:02.902 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-15T04:28:03.476 INFO:teuthology.orchestra.run.smithi047.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T04:28:03.820 INFO:teuthology.orchestra.run.smithi047.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T04:28:03.820 INFO:teuthology.orchestra.run.smithi047.stderr:Extracting ceph user uid/gid from container image... 2023-11-15T04:28:03.820 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-15T04:28:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:stat: 167 167 2023-11-15T04:28:04.423 INFO:teuthology.orchestra.run.smithi047.stderr:Creating initial keys... 2023-11-15T04:28:04.423 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T04:28:04.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-authtool: AQDUSFRl2PikKRAAbdUUq1xIysXLJbKnU1VarQ== 2023-11-15T04:28:05.040 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T04:28:05.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-authtool: AQDVSFRlG9dFEhAA4V97NIBbjW7yUmrGPbbAyw== 2023-11-15T04:28:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T04:28:05.936 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-authtool: AQDVSFRldq2/NxAA1n/dL2zQmjxZXyyvcyz+Bg== 2023-11-15T04:28:06.277 INFO:teuthology.orchestra.run.smithi047.stderr:Creating initial monmap... 2023-11-15T04:28:06.277 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpo6rsh4_5:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 --addv smithi047 [v2:172.21.15.47:3300,v1:172.21.15.47:6789] /tmp/monmap 2023-11-15T04:28:06.574 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T04:28:06.574 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:06.574 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T04:28:06.917 INFO:teuthology.orchestra.run.smithi047.stderr:monmaptool for smithi047 [v2:172.21.15.47:3300,v1:172.21.15.47:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T04:28:06.918 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: set fsid to 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:06.918 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T04:28:06.918 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:28:06.918 INFO:teuthology.orchestra.run.smithi047.stderr:Creating mon... 2023-11-15T04:28:06.919 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/mon.smithi047:/var/lib/ceph/mon/ceph-smithi047:z -v /tmp/ceph-tmpco2izx81:/tmp/keyring:z -v /tmp/ceph-tmpo6rsh4_5:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi047 --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-15T04:28:07.205 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.203+0000 7fc89ed54700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T04:28:07.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.204+0000 7fc89ed54700 1 imported monmap: 2023-11-15T04:28:07.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-15T04:28:07.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:07.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: last_changed 2023-11-15T04:28:06.573922+0000 2023-11-15T04:28:07.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: created 2023-11-15T04:28:06.573922+0000 2023-11-15T04:28:07.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-15T04:28:07.232 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-15T04:28:07.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.smithi047 2023-11-15T04:28:07.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.204+0000 7fc89ed54700 0 /usr/bin/ceph-mon: set fsid to 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:07.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T04:28:07.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T04:28:07.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Compile date May 13 2021 2023-11-15T04:28:07.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: DB SUMMARY 2023-11-15T04:28:07.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi047/store.db dir, Total Num: 0, files: 2023-11-15T04:28:07.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi047/store.db: 2023-11-15T04:28:07.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T04:28:07.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.create_if_missing: 1 2023-11-15T04:28:07.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T04:28:07.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.env: 0x55cde7cf61c0 2023-11-15T04:28:07.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.fs: Posix File System 2023-11-15T04:28:07.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.info_log: 0x55cde8d4eec0 2023-11-15T04:28:07.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T04:28:07.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.statistics: (nil) 2023-11-15T04:28:07.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.use_fsync: 0 2023-11-15T04:28:07.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T04:28:07.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T04:28:07.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T04:28:07.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T04:28:07.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T04:28:07.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T04:28:07.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T04:28:07.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T04:28:07.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T04:28:07.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T04:28:07.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T04:28:07.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.db_log_dir: 2023-11-15T04:28:07.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi047/store.db 2023-11-15T04:28:07.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T04:28:07.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T04:28:07.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T04:28:07.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T04:28:07.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T04:28:07.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T04:28:07.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T04:28:07.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T04:28:07.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T04:28:07.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T04:28:07.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.write_buffer_manager: 0x55cde8d55530 2023-11-15T04:28:07.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T04:28:07.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T04:28:07.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T04:28:07.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T04:28:07.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T04:28:07.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T04:28:07.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T04:28:07.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T04:28:07.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T04:28:07.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.unordered_write: 0 2023-11-15T04:28:07.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T04:28:07.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T04:28:07.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T04:28:07.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T04:28:07.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.row_cache: None 2023-11-15T04:28:07.245 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.wal_filter: None 2023-11-15T04:28:07.245 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T04:28:07.245 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T04:28:07.245 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T04:28:07.245 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T04:28:07.246 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T04:28:07.246 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T04:28:07.246 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T04:28:07.246 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T04:28:07.246 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T04:28:07.247 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T04:28:07.247 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T04:28:07.247 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T04:28:07.247 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T04:28:07.247 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T04:28:07.248 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T04:28:07.248 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T04:28:07.248 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T04:28:07.248 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T04:28:07.248 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T04:28:07.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T04:28:07.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T04:28:07.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.max_open_files: -1 2023-11-15T04:28:07.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T04:28:07.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T04:28:07.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T04:28:07.250 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T04:28:07.250 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Compression algorithms supported: 2023-11-15T04:28:07.250 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T04:28:07.250 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: kZSTD supported: 0 2023-11-15T04:28:07.250 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T04:28:07.251 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T04:28:07.251 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T04:28:07.251 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T04:28:07.251 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T04:28:07.251 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T04:28:07.251 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.205+0000 7fc89ed54700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T04:28:07.252 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-15T04:28:07.252 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.252 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000001 2023-11-15T04:28:07.252 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.252 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T04:28:07.253 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.253 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T04:28:07.253 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.merge_operator: 2023-11-15T04:28:07.253 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_filter: None 2023-11-15T04:28:07.253 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T04:28:07.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T04:28:07.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T04:28:07.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cde8c66120) 2023-11-15T04:28:07.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-15T04:28:07.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T04:28:07.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T04:28:07.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-15T04:28:07.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-15T04:28:07.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-15T04:28:07.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-15T04:28:07.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-15T04:28:07.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-15T04:28:07.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-15T04:28:07.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-15T04:28:07.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_cache: 0x55cde8c9cd10 2023-11-15T04:28:07.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-15T04:28:07.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-15T04:28:07.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-15T04:28:07.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-15T04:28:07.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-15T04:28:07.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-15T04:28:07.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-15T04:28:07.258 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-15T04:28:07.258 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-15T04:28:07.258 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-15T04:28:07.258 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-15T04:28:07.258 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-15T04:28:07.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-15T04:28:07.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-15T04:28:07.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-15T04:28:07.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T04:28:07.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-15T04:28:07.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-15T04:28:07.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-15T04:28:07.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-15T04:28:07.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-15T04:28:07.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-15T04:28:07.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T04:28:07.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T04:28:07.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compression: NoCompression 2023-11-15T04:28:07.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T04:28:07.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T04:28:07.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T04:28:07.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.num_levels: 7 2023-11-15T04:28:07.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T04:28:07.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T04:28:07.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T04:28:07.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T04:28:07.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T04:28:07.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T04:28:07.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T04:28:07.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:28:07.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T04:28:07.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T04:28:07.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T04:28:07.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T04:28:07.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T04:28:07.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:28:07.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T04:28:07.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T04:28:07.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T04:28:07.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T04:28:07.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T04:28:07.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T04:28:07.266 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T04:28:07.266 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T04:28:07.266 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T04:28:07.266 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T04:28:07.266 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T04:28:07.267 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T04:28:07.267 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T04:28:07.267 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T04:28:07.267 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T04:28:07.267 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T04:28:07.267 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T04:28:07.268 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T04:28:07.268 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T04:28:07.268 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T04:28:07.268 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T04:28:07.268 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T04:28:07.269 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T04:28:07.269 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T04:28:07.269 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T04:28:07.269 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T04:28:07.269 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T04:28:07.270 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T04:28:07.270 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T04:28:07.270 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T04:28:07.270 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T04:28:07.270 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T04:28:07.270 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T04:28:07.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T04:28:07.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T04:28:07.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T04:28:07.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T04:28:07.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T04:28:07.272 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T04:28:07.272 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T04:28:07.272 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T04:28:07.272 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T04:28:07.272 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T04:28:07.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T04:28:07.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T04:28:07.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.ttl: 2592000 2023-11-15T04:28:07.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.206+0000 7fc89ed54700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T04:28:07.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.207+0000 7fc89ed54700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T04:28:07.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.207+0000 7fc89ed54700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-15T04:28:07.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.208+0000 7fc89ed54700 4 rocksdb: DB pointer 0x55cde8d61800 2023-11-15T04:28:07.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.208+0000 7fc887bed700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T04:28:07.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.208+0000 7fc887bed700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T04:28:07.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-15T04:28:07.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:28:07.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T04:28:07.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T04:28:07.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:28:07.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T04:28:07.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T04:28:07.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:28:07.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:28:07.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:28:07.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:28:07.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:28:07.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:28:07.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:28:07.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:28:07.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:28:07.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:28:07.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:28:07.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:28:07.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:28:07.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:28:07.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:28:07.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:28:07.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:28:07.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:28:07.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:28:07.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:28:07.281 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:28:07.281 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:28:07.281 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:28:07.281 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:28:07.281 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.282 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:28:07.282 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:28:07.282 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:28:07.282 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:28:07.282 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:28:07.282 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:28:07.283 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:28:07.283 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:28:07.283 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:28:07.283 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:28:07.283 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:28:07.284 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:28:07.284 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.284 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:28:07.284 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2023-11-15T04:28:07.284 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.210+0000 7fc89ed54700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-15T04:28:07.284 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.210+0000 7fc89ed54700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-15T04:28:07.285 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:28:07.210+0000 7fc89ed54700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi047 for mon.smithi047 2023-11-15T04:28:07.564 INFO:teuthology.orchestra.run.smithi047.stderr:create mon.smithi047 on 2023-11-15T04:28:07.565 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:07.571 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl disable ceph.target 2023-11-15T04:28:07.580 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-15T04:28:07.677 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl enable ceph.target 2023-11-15T04:28:07.686 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-15T04:28:07.774 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl start ceph.target 2023-11-15T04:28:07.783 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl enable ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.target 2023-11-15T04:28:07.789 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.target. 2023-11-15T04:28:07.789 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.target. 2023-11-15T04:28:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl start ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.target 2023-11-15T04:28:07.886 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl daemon-reload 2023-11-15T04:28:07.983 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl stop ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047 2023-11-15T04:28:08.012 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl reset-failed ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047 2023-11-15T04:28:08.018 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to reset failed state of unit ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047.service: Unit ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047.service not loaded. 2023-11-15T04:28:08.019 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl enable ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047 2023-11-15T04:28:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.target.wants/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047.service → /etc/systemd/system/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T04:28:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl start ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047 2023-11-15T04:28:08.166 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:08 smithi047 systemd[1]: Starting Ceph mon.smithi047 for 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7... 2023-11-15T04:28:08.465 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:28:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2023-11-15T04:28:08.476 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:28:08.485 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2023-11-15T04:28:08.486 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2023-11-15T04:28:08.487 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T04:28:08.487 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T04:28:08.487 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mon to start... 2023-11-15T04:28:08.488 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mon... 2023-11-15T04:28:08.488 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/mon.smithi047:/var/lib/ceph/mon/ceph-smithi047:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-15T04:28:08.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:08 smithi047 bash[111952]: 897f3c48e1827d46c7c859aedb63cf1bead59ed71073a2ac6432d462f3e50920 2023-11-15T04:28:08.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:08 smithi047 systemd[1]: Started Ceph mon.smithi047 for 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7. 2023-11-15T04:28:08.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: cluster 2023-11-15T04:28:08.486199+0000 mon.smithi047 (mon.0) 0 : [INF] mkfs 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:08.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: cluster 2023-11-15T04:28:08.480762+0000 mon.smithi047 (mon.0) 1 : cluster [INF] mon.smithi047 is new leader, mons smithi047 in quorum (ranks 0) 2023-11-15T04:28:08.924 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.923+0000 7f428b436700 1 Processor -- start 2023-11-15T04:28:08.925 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.923+0000 7f428b436700 1 -- start start 2023-11-15T04:28:08.925 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.923+0000 7f428b436700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f42840fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:08.925 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.923+0000 7f428b436700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42840fec60 con 0x7f42840703f0 2023-11-15T04:28:08.927 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.924+0000 7f428a434700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f42840fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:08.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.924+0000 7f428a434700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f42840fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52426/0 (socket says 172.21.15.47:52426) 2023-11-15T04:28:08.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.924+0000 7f428a434700 1 -- 172.21.15.47:0/3521095332 learned_addr learned my addr 172.21.15.47:0/3521095332 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:08.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.924+0000 7f428a434700 1 -- 172.21.15.47:0/3521095332 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840feda0 con 0x7f42840703f0 2023-11-15T04:28:08.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f428a434700 1 --2- 172.21.15.47:0/3521095332 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f42840fe770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f428000b5f0 tx=0x7f4280006ee0).ready entity=mon.0 client_cookie=4089deb634d2cb68 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:08.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f4289432700 1 -- 172.21.15.47:0/3521095332 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f428000d600 con 0x7f42840703f0 2023-11-15T04:28:08.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f4289432700 1 -- 172.21.15.47:0/3521095332 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428000d760 con 0x7f42840703f0 2023-11-15T04:28:08.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f4289432700 1 -- 172.21.15.47:0/3521095332 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f428000da30 con 0x7f42840703f0 2023-11-15T04:28:08.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f428b436700 1 -- 172.21.15.47:0/3521095332 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 msgr2=0x7f42840fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:08.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f428b436700 1 --2- 172.21.15.47:0/3521095332 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f42840fe770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f428000b5f0 tx=0x7f4280006ee0).stop 2023-11-15T04:28:08.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f428b436700 1 -- 172.21.15.47:0/3521095332 shutdown_connections 2023-11-15T04:28:08.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f428b436700 1 --2- 172.21.15.47:0/3521095332 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f42840fe770 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:08.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f428b436700 1 -- 172.21.15.47:0/3521095332 >> 172.21.15.47:0/3521095332 conn(0x7f42840f6210 msgr2=0x7f42840f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:08.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f428b436700 1 -- 172.21.15.47:0/3521095332 shutdown_connections 2023-11-15T04:28:08.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.925+0000 7f428b436700 1 -- 172.21.15.47:0/3521095332 wait complete. 2023-11-15T04:28:08.931 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.926+0000 7f428b436700 1 Processor -- start 2023-11-15T04:28:08.931 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.926+0000 7f428b436700 1 -- start start 2023-11-15T04:28:08.931 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.926+0000 7f428b436700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f428410a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:08.931 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.926+0000 7f428b436700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428410a5f0 con 0x7f42840703f0 2023-11-15T04:28:08.931 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.926+0000 7f428a434700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f428410a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:08.932 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.926+0000 7f428a434700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f428410a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52430/0 (socket says 172.21.15.47:52430) 2023-11-15T04:28:08.932 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.926+0000 7f428a434700 1 -- 172.21.15.47:0/556701366 learned_addr learned my addr 172.21.15.47:0/556701366 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:08.933 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.927+0000 7f428a434700 1 -- 172.21.15.47:0/556701366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428000b200 con 0x7f42840703f0 2023-11-15T04:28:08.933 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.927+0000 7f428a434700 1 --2- 172.21.15.47:0/556701366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f428410a100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4280006c70 tx=0x7f4280016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:08.933 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.927+0000 7f4272ffd700 1 -- 172.21.15.47:0/556701366 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4280011b40 con 0x7f42840703f0 2023-11-15T04:28:08.934 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.927+0000 7f4272ffd700 1 -- 172.21.15.47:0/556701366 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4280011ca0 con 0x7f42840703f0 2023-11-15T04:28:08.934 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.927+0000 7f428b436700 1 -- 172.21.15.47:0/556701366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428410a7f0 con 0x7f42840703f0 2023-11-15T04:28:08.934 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.927+0000 7f4272ffd700 1 -- 172.21.15.47:0/556701366 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f428001d430 con 0x7f42840703f0 2023-11-15T04:28:08.934 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.927+0000 7f428b436700 1 -- 172.21.15.47:0/556701366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428410ac00 con 0x7f42840703f0 2023-11-15T04:28:08.934 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.927+0000 7f4272ffd700 1 -- 172.21.15.47:0/556701366 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f428000f070 con 0x7f42840703f0 2023-11-15T04:28:08.935 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.927+0000 7f4272ffd700 1 -- 172.21.15.47:0/556701366 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4280027460 con 0x7f42840703f0 2023-11-15T04:28:08.935 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.929+0000 7f428b436700 1 -- 172.21.15.47:0/556701366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4284045ad0 con 0x7f42840703f0 2023-11-15T04:28:08.935 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:08.931+0000 7f4272ffd700 1 -- 172.21.15.47:0/556701366 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4280016070 con 0x7f42840703f0 2023-11-15T04:28:09.008 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.005+0000 7f428b436700 1 -- 172.21.15.47:0/556701366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4284103c40 con 0x7f42840703f0 2023-11-15T04:28:09.008 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.005+0000 7f4272ffd700 1 -- 172.21.15.47:0/556701366 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f4280036070 con 0x7f42840703f0 2023-11-15T04:28:09.008 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: cluster: 2023-11-15T04:28:09.009 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: id: 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:09.009 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-15T04:28:09.009 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.009 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: services: 2023-11-15T04:28:09.009 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi047 (age 0.520845s) 2023-11-15T04:28:09.010 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-15T04:28:09.010 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-15T04:28:09.010 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.010 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: data: 2023-11-15T04:28:09.011 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-15T04:28:09.011 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-15T04:28:09.011 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-15T04:28:09.011 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: pgs: 2023-11-15T04:28:09.011 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.011 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.006+0000 7f428b436700 1 -- 172.21.15.47:0/556701366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 msgr2=0x7f428410a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:09.012 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.006+0000 7f428b436700 1 --2- 172.21.15.47:0/556701366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f428410a100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4280006c70 tx=0x7f4280016000).stop 2023-11-15T04:28:09.012 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.006+0000 7f428b436700 1 -- 172.21.15.47:0/556701366 shutdown_connections 2023-11-15T04:28:09.012 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.006+0000 7f428b436700 1 --2- 172.21.15.47:0/556701366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42840703f0 0x7f428410a100 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:09.012 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.006+0000 7f428b436700 1 -- 172.21.15.47:0/556701366 >> 172.21.15.47:0/556701366 conn(0x7f42840f6210 msgr2=0x7f42840f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:09.012 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.006+0000 7f428b436700 1 -- 172.21.15.47:0/556701366 shutdown_connections 2023-11-15T04:28:09.013 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.006+0000 7f428b436700 1 -- 172.21.15.47:0/556701366 wait complete. 2023-11-15T04:28:09.347 INFO:teuthology.orchestra.run.smithi047.stderr:mon is available 2023-11-15T04:28:09.348 INFO:teuthology.orchestra.run.smithi047.stderr:Assimilating anything we can from ceph.conf... 2023-11-15T04:28:09.348 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z -v /var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/mon.smithi047:/var/lib/ceph/mon/ceph-smithi047:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi047/config 2023-11-15T04:28:09.751 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.749+0000 7fca43cee700 1 Processor -- start 2023-11-15T04:28:09.751 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.749+0000 7fca43cee700 1 -- start start 2023-11-15T04:28:09.751 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.749+0000 7fca43cee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:09.751 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.749+0000 7fca43cee700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0fedd0 con 0x7fca3c070430 2023-11-15T04:28:09.752 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.749+0000 7fca41a8a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:09.752 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.749+0000 7fca41a8a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52444/0 (socket says 172.21.15.47:52444) 2023-11-15T04:28:09.752 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.749+0000 7fca41a8a700 1 -- 172.21.15.47:0/3833483272 learned_addr learned my addr 172.21.15.47:0/3833483272 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:09.752 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca41a8a700 1 -- 172.21.15.47:0/3833483272 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c0fef10 con 0x7fca3c070430 2023-11-15T04:28:09.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca41a8a700 1 --2- 172.21.15.47:0/3833483272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c0fe8e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fca3800b5f0 tx=0x7fca38006ee0).ready entity=mon.0 client_cookie=89582519ce791e4b server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:09.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca40a88700 1 -- 172.21.15.47:0/3833483272 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3800d600 con 0x7fca3c070430 2023-11-15T04:28:09.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca40a88700 1 -- 172.21.15.47:0/3833483272 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3800d760 con 0x7fca3c070430 2023-11-15T04:28:09.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca40a88700 1 -- 172.21.15.47:0/3833483272 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3800da30 con 0x7fca3c070430 2023-11-15T04:28:09.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca43cee700 1 -- 172.21.15.47:0/3833483272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 msgr2=0x7fca3c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:09.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca43cee700 1 --2- 172.21.15.47:0/3833483272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c0fe8e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fca3800b5f0 tx=0x7fca38006ee0).stop 2023-11-15T04:28:09.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca43cee700 1 -- 172.21.15.47:0/3833483272 shutdown_connections 2023-11-15T04:28:09.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca43cee700 1 --2- 172.21.15.47:0/3833483272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c0fe8e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.750+0000 7fca43cee700 1 -- 172.21.15.47:0/3833483272 >> 172.21.15.47:0/3833483272 conn(0x7fca3c0f63c0 msgr2=0x7fca3c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.751+0000 7fca43cee700 1 -- 172.21.15.47:0/3833483272 shutdown_connections 2023-11-15T04:28:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.751+0000 7fca43cee700 1 -- 172.21.15.47:0/3833483272 wait complete. 2023-11-15T04:28:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.751+0000 7fca43cee700 1 Processor -- start 2023-11-15T04:28:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.751+0000 7fca43cee700 1 -- start start 2023-11-15T04:28:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.751+0000 7fca43cee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c10a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.751+0000 7fca43cee700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c10a770 con 0x7fca3c070430 2023-11-15T04:28:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca41a8a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c10a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca41a8a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c10a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52448/0 (socket says 172.21.15.47:52448) 2023-11-15T04:28:09.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca41a8a700 1 -- 172.21.15.47:0/3344000677 learned_addr learned my addr 172.21.15.47:0/3344000677 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:09.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca41a8a700 1 -- 172.21.15.47:0/3344000677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3800b200 con 0x7fca3c070430 2023-11-15T04:28:09.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca41a8a700 1 --2- 172.21.15.47:0/3344000677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c10a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fca38006c70 tx=0x7fca38016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:09.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca2a7fc700 1 -- 172.21.15.47:0/3344000677 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca38011b40 con 0x7fca3c070430 2023-11-15T04:28:09.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca2a7fc700 1 -- 172.21.15.47:0/3344000677 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca38011ca0 con 0x7fca3c070430 2023-11-15T04:28:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca43cee700 1 -- 172.21.15.47:0/3344000677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c10a970 con 0x7fca3c070430 2023-11-15T04:28:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca2a7fc700 1 -- 172.21.15.47:0/3344000677 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3801d430 con 0x7fca3c070430 2023-11-15T04:28:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.752+0000 7fca43cee700 1 -- 172.21.15.47:0/3344000677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c10ad80 con 0x7fca3c070430 2023-11-15T04:28:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.753+0000 7fca2a7fc700 1 -- 172.21.15.47:0/3344000677 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fca3800f070 con 0x7fca3c070430 2023-11-15T04:28:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.753+0000 7fca2a7fc700 1 -- 172.21.15.47:0/3344000677 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fca38027460 con 0x7fca3c070430 2023-11-15T04:28:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.754+0000 7fca43cee700 1 -- 172.21.15.47:0/3344000677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c045ad0 con 0x7fca3c070430 2023-11-15T04:28:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.756+0000 7fca2a7fc700 1 -- 172.21.15.47:0/3344000677 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fca38016070 con 0x7fca3c070430 2023-11-15T04:28:09.762 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: cluster 2023-11-15T04:28:08.485521+0000 mon.smithi047 (mon.0) 2 : cluster [INF] mon.smithi047 is new leader, mons smithi047 in quorum (ranks 0) 2023-11-15T04:28:09.762 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: cluster 2023-11-15T04:28:08.485732+0000 mon.smithi047 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi047=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2023-11-15T04:28:09.762 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: cluster 2023-11-15T04:28:08.487765+0000 mon.smithi047 (mon.0) 4 : cluster [DBG] fsmap 2023-11-15T04:28:09.763 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: cluster 2023-11-15T04:28:08.489654+0000 mon.smithi047 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T04:28:09.763 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: cluster 2023-11-15T04:28:08.490127+0000 mon.smithi047 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T04:28:09.763 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: audit 2023-11-15T04:28:09.006326+0000 mon.smithi047 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.47:0/556701366' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-15T04:28:09.827 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.826+0000 7fca43cee700 1 -- 172.21.15.47:0/3344000677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fca3c0ff160 con 0x7fca3c070430 2023-11-15T04:28:09.831 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.830+0000 7fca2a7fc700 1 -- 172.21.15.47:0/3344000677 <== mon.0 v2:172.21.15.47:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fca3801d660 con 0x7fca3c070430 2023-11-15T04:28:09.831 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.830+0000 7fca2a7fc700 1 -- 172.21.15.47:0/3344000677 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7fca38036020 con 0x7fca3c070430 2023-11-15T04:28:09.831 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.834 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [global] 2023-11-15T04:28:09.834 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: fsid = 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:09.834 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.47:3300,v1:172.21.15.47:6789] 2023-11-15T04:28:09.834 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-15T04:28:09.835 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-15T04:28:09.835 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-15T04:28:09.835 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-15T04:28:09.835 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.835 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [mgr] 2023-11-15T04:28:09.836 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-15T04:28:09.836 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.836 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [osd] 2023-11-15T04:28:09.836 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-15T04:28:09.836 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-15T04:28:09.837 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-15T04:28:09.837 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-15T04:28:09.837 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-15T04:28:09.837 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-15T04:28:09.837 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.831+0000 7fca43cee700 1 -- 172.21.15.47:0/3344000677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 msgr2=0x7fca3c10a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:09.838 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.831+0000 7fca43cee700 1 --2- 172.21.15.47:0/3344000677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c10a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fca38006c70 tx=0x7fca38016000).stop 2023-11-15T04:28:09.838 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.831+0000 7fca43cee700 1 -- 172.21.15.47:0/3344000677 shutdown_connections 2023-11-15T04:28:09.838 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.831+0000 7fca43cee700 1 --2- 172.21.15.47:0/3344000677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca3c070430 0x7fca3c10a280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:09.838 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.831+0000 7fca43cee700 1 -- 172.21.15.47:0/3344000677 >> 172.21.15.47:0/3344000677 conn(0x7fca3c0f63c0 msgr2=0x7fca3c0f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:09.838 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.832+0000 7fca43cee700 1 -- 172.21.15.47:0/3344000677 shutdown_connections 2023-11-15T04:28:09.839 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:09.832+0000 7fca43cee700 1 -- 172.21.15.47:0/3344000677 wait complete. 2023-11-15T04:28:10.146 INFO:teuthology.orchestra.run.smithi047.stderr:Generating new minimal ceph.conf... 2023-11-15T04:28:10.146 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z -v /var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/mon.smithi047:/var/lib/ceph/mon/ceph-smithi047:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi047/config 2023-11-15T04:28:10.498 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.497+0000 7f0957e7c700 1 Processor -- start 2023-11-15T04:28:10.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.497+0000 7f0957e7c700 1 -- start start 2023-11-15T04:28:10.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.497+0000 7f0957e7c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f0950100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:10.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.497+0000 7f0957e7c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09500fc750 con 0x7f09500703b0 2023-11-15T04:28:10.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.497+0000 7f0955c18700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f0950100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:10.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.498+0000 7f0955c18700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f0950100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52454/0 (socket says 172.21.15.47:52454) 2023-11-15T04:28:10.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.498+0000 7f0955c18700 1 -- 172.21.15.47:0/1311131561 learned_addr learned my addr 172.21.15.47:0/1311131561 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:10.504 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.498+0000 7f0955c18700 1 -- 172.21.15.47:0/1311131561 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950101250 con 0x7f09500703b0 2023-11-15T04:28:10.504 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.498+0000 7f0955c18700 1 --2- 172.21.15.47:0/1311131561 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f0950100d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f094c00b5f0 tx=0x7f094c006ee0).ready entity=mon.0 client_cookie=ff1958e935d707cd server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:10.505 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.498+0000 7f0954c16700 1 -- 172.21.15.47:0/1311131561 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f094c00d600 con 0x7f09500703b0 2023-11-15T04:28:10.505 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.498+0000 7f0954c16700 1 -- 172.21.15.47:0/1311131561 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f094c00dbd0 con 0x7f09500703b0 2023-11-15T04:28:10.506 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.498+0000 7f0954c16700 1 -- 172.21.15.47:0/1311131561 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f094c011c00 con 0x7f09500703b0 2023-11-15T04:28:10.506 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.499+0000 7f0957e7c700 1 -- 172.21.15.47:0/1311131561 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 msgr2=0x7f0950100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:10.506 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.499+0000 7f0957e7c700 1 --2- 172.21.15.47:0/1311131561 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f0950100d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f094c00b5f0 tx=0x7f094c006ee0).stop 2023-11-15T04:28:10.506 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.499+0000 7f0957e7c700 1 -- 172.21.15.47:0/1311131561 shutdown_connections 2023-11-15T04:28:10.507 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.499+0000 7f0957e7c700 1 --2- 172.21.15.47:0/1311131561 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f0950100d60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:10.507 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.499+0000 7f0957e7c700 1 -- 172.21.15.47:0/1311131561 >> 172.21.15.47:0/1311131561 conn(0x7f09500f63c0 msgr2=0x7f09500f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:10.507 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.499+0000 7f0957e7c700 1 -- 172.21.15.47:0/1311131561 shutdown_connections 2023-11-15T04:28:10.507 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.499+0000 7f0957e7c700 1 -- 172.21.15.47:0/1311131561 wait complete. 2023-11-15T04:28:10.507 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.499+0000 7f0957e7c700 1 Processor -- start 2023-11-15T04:28:10.508 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.500+0000 7f0957e7c700 1 -- start start 2023-11-15T04:28:10.508 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.500+0000 7f0957e7c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f095010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:10.508 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.500+0000 7f0957e7c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095010d100 con 0x7f09500703b0 2023-11-15T04:28:10.508 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.500+0000 7f0955c18700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f095010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:10.509 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.500+0000 7f0955c18700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f095010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52458/0 (socket says 172.21.15.47:52458) 2023-11-15T04:28:10.509 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.500+0000 7f0955c18700 1 -- 172.21.15.47:0/2840403390 learned_addr learned my addr 172.21.15.47:0/2840403390 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:10.509 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.500+0000 7f0955c18700 1 -- 172.21.15.47:0/2840403390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c00b200 con 0x7f09500703b0 2023-11-15T04:28:10.509 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.500+0000 7f0955c18700 1 --2- 172.21.15.47:0/2840403390 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f095010cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f094c00b190 tx=0x7f094c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:10.509 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.501+0000 7f093e7fc700 1 -- 172.21.15.47:0/2840403390 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f094c01b410 con 0x7f09500703b0 2023-11-15T04:28:10.509 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.501+0000 7f093e7fc700 1 -- 172.21.15.47:0/2840403390 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f094c01b9e0 con 0x7f09500703b0 2023-11-15T04:28:10.510 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.501+0000 7f0957e7c700 1 -- 172.21.15.47:0/2840403390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095010d300 con 0x7f09500703b0 2023-11-15T04:28:10.510 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.501+0000 7f093e7fc700 1 -- 172.21.15.47:0/2840403390 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f094c01a880 con 0x7f09500703b0 2023-11-15T04:28:10.510 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.501+0000 7f0957e7c700 1 -- 172.21.15.47:0/2840403390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095010d710 con 0x7f09500703b0 2023-11-15T04:28:10.510 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.501+0000 7f093e7fc700 1 -- 172.21.15.47:0/2840403390 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f094c00f070 con 0x7f09500703b0 2023-11-15T04:28:10.510 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.501+0000 7f093e7fc700 1 -- 172.21.15.47:0/2840403390 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f094c025c90 con 0x7f09500703b0 2023-11-15T04:28:10.511 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.502+0000 7f0957e7c700 1 -- 172.21.15.47:0/2840403390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0950045ad0 con 0x7f09500703b0 2023-11-15T04:28:10.511 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.505+0000 7f093e7fc700 1 -- 172.21.15.47:0/2840403390 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f094c016070 con 0x7f09500703b0 2023-11-15T04:28:10.576 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.575+0000 7f0957e7c700 1 -- 172.21.15.47:0/2840403390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0950106a30 con 0x7f09500703b0 2023-11-15T04:28:10.577 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.576+0000 7f093e7fc700 1 -- 172.21.15.47:0/2840403390 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f094c01bb50 con 0x7f09500703b0 2023-11-15T04:28:10.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.577+0000 7f0957e7c700 1 -- 172.21.15.47:0/2840403390 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 msgr2=0x7f095010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:10.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.577+0000 7f0957e7c700 1 --2- 172.21.15.47:0/2840403390 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f095010cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f094c00b190 tx=0x7f094c006ee0).stop 2023-11-15T04:28:10.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.577+0000 7f0957e7c700 1 -- 172.21.15.47:0/2840403390 shutdown_connections 2023-11-15T04:28:10.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.577+0000 7f0957e7c700 1 --2- 172.21.15.47:0/2840403390 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09500703b0 0x7f095010cc10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:10.579 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.577+0000 7f0957e7c700 1 -- 172.21.15.47:0/2840403390 >> 172.21.15.47:0/2840403390 conn(0x7f09500f63c0 msgr2=0x7f09500f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:10.579 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.577+0000 7f0957e7c700 1 -- 172.21.15.47:0/2840403390 shutdown_connections 2023-11-15T04:28:10.579 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:10.577+0000 7f0957e7c700 1 -- 172.21.15.47:0/2840403390 wait complete. 2023-11-15T04:28:10.753 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: audit 2023-11-15T04:28:09.828022+0000 mon.smithi047 (mon.0) 8 : audit [INF] from='client.? 172.21.15.47:0/3344000677' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T04:28:10.898 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: audit 2023-11-15T04:28:09.831318+0000 mon.smithi047 (mon.0) 9 : audit [INF] from='client.? 172.21.15.47:0/3344000677' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T04:28:10.956 INFO:teuthology.orchestra.run.smithi047.stderr:Restarting the monitor... 2023-11-15T04:28:10.956 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl restart ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047 2023-11-15T04:28:11.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:10 smithi047 systemd[1]: Stopping Ceph mon.smithi047 for 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7... 2023-11-15T04:28:11.336 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: debug 2023-11-15T04:28:11.036+0000 7f4e5c9fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi047 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T04:28:11.337 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[111968]: debug 2023-11-15T04:28:11.036+0000 7f4e5c9fd700 -1 mon.smithi047@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T04:28:11.337 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 podman[112289]: ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047 2023-11-15T04:28:11.643 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 systemd[1]: ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047.service: Succeeded. 2023-11-15T04:28:11.643 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 systemd[1]: Stopped Ceph mon.smithi047 for 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7. 2023-11-15T04:28:11.644 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 systemd[1]: Starting Ceph mon.smithi047 for 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7... 2023-11-15T04:28:11.647 INFO:teuthology.orchestra.run.smithi047.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-15T04:28:11.649 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-15T04:28:12.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 bash[112362]: 1c9919948e5cc476656f0273df9163752fda0faf354a11ce6ba5ae796217a819 2023-11-15T04:28:12.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 systemd[1]: Started Ceph mon.smithi047 for 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7. 2023-11-15T04:28:12.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.654+0000 7fd978088700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T04:28:12.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.654+0000 7fd978088700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-15T04:28:12.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.654+0000 7fd978088700 0 pidfile_write: ignore empty --pid-file 2023-11-15T04:28:12.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.659+0000 7fd978088700 0 load: jerasure load: lrc load: isa 2023-11-15T04:28:12.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T04:28:12.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T04:28:12.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Compile date May 13 2021 2023-11-15T04:28:12.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: DB SUMMARY 2023-11-15T04:28:12.011 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.011 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: CURRENT file: CURRENT 2023-11-15T04:28:12.011 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.011 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-15T04:28:12.011 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.012 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-15T04:28:12.012 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.012 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi047/store.db dir, Total Num: 1, files: 000004.sst 2023-11-15T04:28:12.012 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.012 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi047/store.db: 000006.log size: 92914 ; 2023-11-15T04:28:12.013 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.013 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T04:28:12.013 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.create_if_missing: 0 2023-11-15T04:28:12.013 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T04:28:12.014 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.env: 0x55bad50ec1c0 2023-11-15T04:28:12.014 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.fs: Posix File System 2023-11-15T04:28:12.014 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.info_log: 0x55bad6613960 2023-11-15T04:28:12.014 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T04:28:12.014 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.statistics: (nil) 2023-11-15T04:28:12.015 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.use_fsync: 0 2023-11-15T04:28:12.015 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T04:28:12.015 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T04:28:12.015 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T04:28:12.015 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T04:28:12.016 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T04:28:12.016 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T04:28:12.016 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T04:28:12.016 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T04:28:12.016 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T04:28:12.017 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T04:28:12.017 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T04:28:12.017 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.db_log_dir: 2023-11-15T04:28:12.017 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi047/store.db 2023-11-15T04:28:12.017 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T04:28:12.018 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T04:28:12.018 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T04:28:12.018 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T04:28:12.018 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T04:28:12.018 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T04:28:12.019 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T04:28:12.019 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T04:28:12.019 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T04:28:12.020 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T04:28:12.020 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.write_buffer_manager: 0x55bad7288030 2023-11-15T04:28:12.020 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T04:28:12.020 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T04:28:12.020 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T04:28:12.021 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T04:28:12.021 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T04:28:12.021 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T04:28:12.021 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T04:28:12.021 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T04:28:12.022 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T04:28:12.022 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.unordered_write: 0 2023-11-15T04:28:12.022 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T04:28:12.022 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T04:28:12.022 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T04:28:12.023 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T04:28:12.023 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.row_cache: None 2023-11-15T04:28:12.023 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.wal_filter: None 2023-11-15T04:28:12.023 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T04:28:12.024 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T04:28:12.024 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T04:28:12.024 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T04:28:12.024 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T04:28:12.024 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T04:28:12.024 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T04:28:12.025 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T04:28:12.025 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T04:28:12.025 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T04:28:12.025 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T04:28:12.025 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T04:28:12.026 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T04:28:12.026 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T04:28:12.026 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T04:28:12.026 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T04:28:12.026 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T04:28:12.027 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T04:28:12.027 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T04:28:12.027 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T04:28:12.027 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T04:28:12.027 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_open_files: -1 2023-11-15T04:28:12.027 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T04:28:12.028 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T04:28:12.028 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T04:28:12.028 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T04:28:12.028 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Compression algorithms supported: 2023-11-15T04:28:12.028 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T04:28:12.029 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: kZSTD supported: 0 2023-11-15T04:28:12.029 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T04:28:12.029 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T04:28:12.029 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T04:28:12.029 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T04:28:12.030 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T04:28:12.030 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T04:28:12.030 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T04:28:12.030 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000005 2023-11-15T04:28:12.031 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.031 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T04:28:12.031 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.031 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T04:28:12.031 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.merge_operator: 2023-11-15T04:28:12.032 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_filter: None 2023-11-15T04:28:12.032 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T04:28:12.032 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T04:28:12.032 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T04:28:12.032 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bad652a128) 2023-11-15T04:28:12.033 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cache_index_and_filter_blocks: 1 2023-11-15T04:28:12.033 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T04:28:12.033 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T04:28:12.033 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: pin_top_level_index_and_filter: 1 2023-11-15T04:28:12.033 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: index_type: 0 2023-11-15T04:28:12.034 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: data_block_index_type: 0 2023-11-15T04:28:12.034 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: index_shortening: 1 2023-11-15T04:28:12.034 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: data_block_hash_table_util_ratio: 0.750000 2023-11-15T04:28:12.034 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: hash_index_allow_collision: 1 2023-11-15T04:28:12.034 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: checksum: 1 2023-11-15T04:28:12.035 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: no_block_cache: 0 2023-11-15T04:28:12.035 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: block_cache: 0x55bad6560f10 2023-11-15T04:28:12.035 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: block_cache_name: BinnedLRUCache 2023-11-15T04:28:12.036 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: block_cache_options: 2023-11-15T04:28:12.036 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: capacity : 536870912 2023-11-15T04:28:12.038 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: num_shard_bits : 4 2023-11-15T04:28:12.039 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: strict_capacity_limit : 0 2023-11-15T04:28:12.039 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: high_pri_pool_ratio: 0.000 2023-11-15T04:28:12.039 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: block_cache_compressed: (nil) 2023-11-15T04:28:12.039 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: persistent_cache: (nil) 2023-11-15T04:28:12.039 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: block_size: 4096 2023-11-15T04:28:12.040 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: block_size_deviation: 10 2023-11-15T04:28:12.040 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: block_restart_interval: 16 2023-11-15T04:28:12.040 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: index_block_restart_interval: 1 2023-11-15T04:28:12.040 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: metadata_block_size: 4096 2023-11-15T04:28:12.040 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: partition_filters: 0 2023-11-15T04:28:12.041 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: use_delta_encoding: 1 2023-11-15T04:28:12.041 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T04:28:12.041 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: whole_key_filtering: 1 2023-11-15T04:28:12.041 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: verify_compression: 0 2023-11-15T04:28:12.042 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: read_amp_bytes_per_bit: 0 2023-11-15T04:28:12.042 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: format_version: 2 2023-11-15T04:28:12.042 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: enable_index_compression: 1 2023-11-15T04:28:12.042 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: block_align: 0 2023-11-15T04:28:12.042 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.043 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T04:28:12.043 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T04:28:12.043 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compression: NoCompression 2023-11-15T04:28:12.043 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T04:28:12.043 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T04:28:12.044 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T04:28:12.044 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.num_levels: 7 2023-11-15T04:28:12.044 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T04:28:12.044 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T04:28:12.044 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T04:28:12.044 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T04:28:12.045 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T04:28:12.045 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T04:28:12.045 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T04:28:12.045 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:28:12.046 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T04:28:12.046 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T04:28:12.046 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T04:28:12.046 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T04:28:12.046 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T04:28:12.047 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:28:12.047 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T04:28:12.047 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T04:28:12.047 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T04:28:12.047 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T04:28:12.047 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T04:28:12.048 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T04:28:12.048 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T04:28:12.048 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T04:28:12.048 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T04:28:12.049 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T04:28:12.049 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T04:28:12.049 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T04:28:12.049 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T04:28:12.049 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T04:28:12.049 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T04:28:12.050 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T04:28:12.050 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T04:28:12.050 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T04:28:12.050 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T04:28:12.050 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T04:28:12.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T04:28:12.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T04:28:12.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T04:28:12.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T04:28:12.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T04:28:12.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T04:28:12.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T04:28:12.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T04:28:12.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T04:28:12.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T04:28:12.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T04:28:12.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T04:28:12.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T04:28:12.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T04:28:12.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T04:28:12.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T04:28:12.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T04:28:12.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T04:28:12.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T04:28:12.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T04:28:12.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T04:28:12.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T04:28:12.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T04:28:12.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T04:28:12.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T04:28:12.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.ttl: 2592000 2023-11-15T04:28:12.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.660+0000 7fd978088700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T04:28:12.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.661+0000 7fd978088700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T04:28:12.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.661+0000 7fd978088700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-15T04:28:12.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.661+0000 7fd978088700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700022491662927, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-15T04:28:12.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.661+0000 7fd978088700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-15T04:28:12.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.662+0000 7fd978088700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-15T04:28:12.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.662+0000 7fd978088700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700022491663984, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88694, "table_properties": {"data_size": 86809, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79950, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700022491, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-15T04:28:12.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.663+0000 7fd978088700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-15T04:28:12.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.663+0000 7fd978088700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700022491664480, "job": 1, "event": "recovery_finished"} 2023-11-15T04:28:12.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.664+0000 7fd978088700 4 rocksdb: DB pointer 0x55bad6625800 2023-11-15T04:28:12.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.664+0000 7fd95eea0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T04:28:12.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.664+0000 7fd95eea0700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T04:28:12.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ** DB Stats ** 2023-11-15T04:28:12.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:28:12.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T04:28:12.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T04:28:12.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:28:12.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T04:28:12.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T04:28:12.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:28:12.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ** Compaction Stats [default] ** 2023-11-15T04:28:12.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:28:12.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:28:12.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2023-11-15T04:28:12.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2023-11-15T04:28:12.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2023-11-15T04:28:12.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ** Compaction Stats [default] ** 2023-11-15T04:28:12.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:28:12.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:28:12.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2023-11-15T04:28:12.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:28:12.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:28:12.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:28:12.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:28:12.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:28:12.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:28:12.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Cumulative compaction: 0.00 GB write, 23.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:28:12.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Interval compaction: 0.00 GB write, 23.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:28:12.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:28:12.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:28:12.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ** Compaction Stats [default] ** 2023-11-15T04:28:12.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:28:12.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:28:12.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2023-11-15T04:28:12.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2023-11-15T04:28:12.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:28:12.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ** Compaction Stats [default] ** 2023-11-15T04:28:12.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:28:12.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:28:12.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2023-11-15T04:28:12.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:28:12.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:28:12.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:28:12.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:28:12.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:28:12.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:28:12.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Cumulative compaction: 0.00 GB write, 23.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:28:12.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:28:12.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:28:12.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:28:12.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: 2023-11-15T04:28:12.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.665+0000 7fd978088700 0 starting mon.smithi047 rank 0 at public addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] at bind addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi047 fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:12.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:28:11.666+0000 7fd978088700 1 mon.smithi047@-1(???) e1 preinit fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:12.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:09.828022+0000 mon.smithi047 (mon.0) 8 : audit [INF] from='client.? 172.21.15.47:0/3344000677' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T04:28:12.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:09.831318+0000 mon.smithi047 (mon.0) 9 : audit [INF] from='client.? 172.21.15.47:0/3344000677' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T04:28:12.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:11.684034+0000 mon.smithi047 (mon.0) 1 : cluster [INF] mon.smithi047 is new leader, mons smithi047 in quorum (ranks 0) 2023-11-15T04:28:12.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:11.684137+0000 mon.smithi047 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi047=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2023-11-15T04:28:12.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:11.684197+0000 mon.smithi047 (mon.0) 3 : cluster [DBG] fsmap 2023-11-15T04:28:12.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:11.684254+0000 mon.smithi047 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T04:28:12.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:11.684906+0000 mon.smithi047 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T04:28:12.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.020+0000 7fe6e06f7700 1 Processor -- start 2023-11-15T04:28:12.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.020+0000 7fe6e06f7700 1 -- start start 2023-11-15T04:28:12.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.020+0000 7fe6e06f7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d80fe9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:12.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.020+0000 7fe6e06f7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d8070770 con 0x7fe6d8070350 2023-11-15T04:28:12.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.021+0000 7fe6de493700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d80fe9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:12.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.021+0000 7fe6de493700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d80fe9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52472/0 (socket says 172.21.15.47:52472) 2023-11-15T04:28:12.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.021+0000 7fe6de493700 1 -- 172.21.15.47:0/837782365 learned_addr learned my addr 172.21.15.47:0/837782365 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:12.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.021+0000 7fe6de493700 1 -- 172.21.15.47:0/837782365 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80feed0 con 0x7fe6d8070350 2023-11-15T04:28:12.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.021+0000 7fe6de493700 1 --2- 172.21.15.47:0/837782365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d80fe9e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe6d400b5f0 tx=0x7fe6d4006ee0).ready entity=mon.0 client_cookie=72b859d67f44796c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:12.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.021+0000 7fe6dd491700 1 -- 172.21.15.47:0/837782365 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d400d600 con 0x7fe6d8070350 2023-11-15T04:28:12.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.021+0000 7fe6dd491700 1 -- 172.21.15.47:0/837782365 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe6d400dbd0 con 0x7fe6d8070350 2023-11-15T04:28:12.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.022+0000 7fe6dd491700 1 -- 172.21.15.47:0/837782365 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d4011c00 con 0x7fe6d8070350 2023-11-15T04:28:12.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.022+0000 7fe6e06f7700 1 -- 172.21.15.47:0/837782365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 msgr2=0x7fe6d80fe9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:12.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.022+0000 7fe6e06f7700 1 --2- 172.21.15.47:0/837782365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d80fe9e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe6d400b5f0 tx=0x7fe6d4006ee0).stop 2023-11-15T04:28:12.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.022+0000 7fe6e06f7700 1 -- 172.21.15.47:0/837782365 shutdown_connections 2023-11-15T04:28:12.081 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.022+0000 7fe6e06f7700 1 --2- 172.21.15.47:0/837782365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d80fe9e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:12.081 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.022+0000 7fe6e06f7700 1 -- 172.21.15.47:0/837782365 >> 172.21.15.47:0/837782365 conn(0x7fe6d80f63c0 msgr2=0x7fe6d80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:12.081 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.022+0000 7fe6e06f7700 1 -- 172.21.15.47:0/837782365 shutdown_connections 2023-11-15T04:28:12.081 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.022+0000 7fe6e06f7700 1 -- 172.21.15.47:0/837782365 wait complete. 2023-11-15T04:28:12.081 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.023+0000 7fe6e06f7700 1 Processor -- start 2023-11-15T04:28:12.081 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.023+0000 7fe6e06f7700 1 -- start start 2023-11-15T04:28:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.023+0000 7fe6e06f7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.023+0000 7fe6e06f7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d810aee0 con 0x7fe6d8070350 2023-11-15T04:28:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.023+0000 7fe6de493700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.023+0000 7fe6de493700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52476/0 (socket says 172.21.15.47:52476) 2023-11-15T04:28:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.023+0000 7fe6de493700 1 -- 172.21.15.47:0/170803575 learned_addr learned my addr 172.21.15.47:0/170803575 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.024+0000 7fe6de493700 1 -- 172.21.15.47:0/170803575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d400b200 con 0x7fe6d8070350 2023-11-15T04:28:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.024+0000 7fe6de493700 1 --2- 172.21.15.47:0/170803575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d810a9f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe6d400e040 tx=0x7fe6d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.024+0000 7fe6c6ffd700 1 -- 172.21.15.47:0/170803575 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d400d9f0 con 0x7fe6d8070350 2023-11-15T04:28:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.024+0000 7fe6c6ffd700 1 -- 172.21.15.47:0/170803575 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe6d401a9f0 con 0x7fe6d8070350 2023-11-15T04:28:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.024+0000 7fe6c6ffd700 1 -- 172.21.15.47:0/170803575 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d401b890 con 0x7fe6d8070350 2023-11-15T04:28:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.024+0000 7fe6e06f7700 1 -- 172.21.15.47:0/170803575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d810b0e0 con 0x7fe6d8070350 2023-11-15T04:28:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.024+0000 7fe6e06f7700 1 -- 172.21.15.47:0/170803575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d810b4f0 con 0x7fe6d8070350 2023-11-15T04:28:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.024+0000 7fe6c6ffd700 1 -- 172.21.15.47:0/170803575 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe6d400f070 con 0x7fe6d8070350 2023-11-15T04:28:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.024+0000 7fe6c6ffd700 1 -- 172.21.15.47:0/170803575 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe6d4025b30 con 0x7fe6d8070350 2023-11-15T04:28:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.025+0000 7fe6e06f7700 1 -- 172.21.15.47:0/170803575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d8045ad0 con 0x7fe6d8070350 2023-11-15T04:28:12.085 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.029+0000 7fe6c6ffd700 1 -- 172.21.15.47:0/170803575 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe6d4016070 con 0x7fe6d8070350 2023-11-15T04:28:12.099 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.098+0000 7fe6e06f7700 1 -- 172.21.15.47:0/170803575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fe6d81048c0 con 0x7fe6d8070350 2023-11-15T04:28:12.101 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.101+0000 7fe6c6ffd700 1 -- 172.21.15.47:0/170803575 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe6d4025cf0 con 0x7fe6d8070350 2023-11-15T04:28:12.102 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.101+0000 7fe6e06f7700 1 -- 172.21.15.47:0/170803575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 msgr2=0x7fe6d810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:12.103 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.101+0000 7fe6e06f7700 1 --2- 172.21.15.47:0/170803575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d810a9f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe6d400e040 tx=0x7fe6d4006ee0).stop 2023-11-15T04:28:12.103 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.101+0000 7fe6e06f7700 1 -- 172.21.15.47:0/170803575 shutdown_connections 2023-11-15T04:28:12.103 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.101+0000 7fe6e06f7700 1 --2- 172.21.15.47:0/170803575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6d8070350 0x7fe6d810a9f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:12.103 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.101+0000 7fe6e06f7700 1 -- 172.21.15.47:0/170803575 >> 172.21.15.47:0/170803575 conn(0x7fe6d80f63c0 msgr2=0x7fe6d80f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:12.104 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.101+0000 7fe6e06f7700 1 -- 172.21.15.47:0/170803575 shutdown_connections 2023-11-15T04:28:12.104 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:12.101+0000 7fe6e06f7700 1 -- 172.21.15.47:0/170803575 wait complete. 2023-11-15T04:28:12.446 INFO:teuthology.orchestra.run.smithi047.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-15T04:28:12.447 INFO:teuthology.orchestra.run.smithi047.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-15T04:28:12.447 INFO:teuthology.orchestra.run.smithi047.stderr:Creating mgr... 2023-11-15T04:28:12.448 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying port 9283 ... 2023-11-15T04:28:12.450 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl daemon-reload 2023-11-15T04:28:12.561 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl stop ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mgr.smithi047.zgysuz 2023-11-15T04:28:12.588 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl reset-failed ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mgr.smithi047.zgysuz 2023-11-15T04:28:12.594 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to reset failed state of unit ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mgr.smithi047.zgysuz.service: Unit ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mgr.smithi047.zgysuz.service not loaded. 2023-11-15T04:28:12.595 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl enable ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mgr.smithi047.zgysuz 2023-11-15T04:28:12.601 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.target.wants/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mgr.smithi047.zgysuz.service → /etc/systemd/system/ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T04:28:12.699 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl start ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mgr.smithi047.zgysuz 2023-11-15T04:28:13.015 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:28:13.026 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2023-11-15T04:28:13.027 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:28:13.035 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2023-11-15T04:28:13.036 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2023-11-15T04:28:13.037 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T04:28:13.037 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T04:28:13.037 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:28:13.045 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2023-11-15T04:28:13.046 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:28:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2023-11-15T04:28:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2023-11-15T04:28:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-15T04:28:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr to start... 2023-11-15T04:28:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr... 2023-11-15T04:28:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:28:13.459 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.456+0000 7f1badffc700 1 Processor -- start 2023-11-15T04:28:13.459 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.456+0000 7f1badffc700 1 -- start start 2023-11-15T04:28:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.457+0000 7f1badffc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba8100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.457+0000 7f1badffc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba80fc750 con 0x7f1ba80703b0 2023-11-15T04:28:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.457+0000 7f1ba77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba8100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.457+0000 7f1ba77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba8100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52518/0 (socket says 172.21.15.47:52518) 2023-11-15T04:28:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.457+0000 7f1ba77fe700 1 -- 172.21.15.47:0/3763545532 learned_addr learned my addr 172.21.15.47:0/3763545532 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:13.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.458+0000 7f1ba77fe700 1 -- 172.21.15.47:0/3763545532 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba8101250 con 0x7f1ba80703b0 2023-11-15T04:28:13.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.458+0000 7f1ba77fe700 1 --2- 172.21.15.47:0/3763545532 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba8100d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1b9800b5f0 tx=0x7f1b98006ee0).ready entity=mon.0 client_cookie=acde2447a4cc209 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:13.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.458+0000 7f1ba67fc700 1 -- 172.21.15.47:0/3763545532 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9800d600 con 0x7f1ba80703b0 2023-11-15T04:28:13.464 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.458+0000 7f1ba67fc700 1 -- 172.21.15.47:0/3763545532 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1b9800dbd0 con 0x7f1ba80703b0 2023-11-15T04:28:13.464 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.458+0000 7f1ba67fc700 1 -- 172.21.15.47:0/3763545532 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b98011c00 con 0x7f1ba80703b0 2023-11-15T04:28:13.464 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.459+0000 7f1badffc700 1 -- 172.21.15.47:0/3763545532 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 msgr2=0x7f1ba8100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:13.464 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.459+0000 7f1badffc700 1 --2- 172.21.15.47:0/3763545532 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba8100d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1b9800b5f0 tx=0x7f1b98006ee0).stop 2023-11-15T04:28:13.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.459+0000 7f1badffc700 1 -- 172.21.15.47:0/3763545532 shutdown_connections 2023-11-15T04:28:13.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.459+0000 7f1badffc700 1 --2- 172.21.15.47:0/3763545532 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba8100d60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:13.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.459+0000 7f1badffc700 1 -- 172.21.15.47:0/3763545532 >> 172.21.15.47:0/3763545532 conn(0x7f1ba80f63c0 msgr2=0x7f1ba80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:13.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.459+0000 7f1badffc700 1 -- 172.21.15.47:0/3763545532 shutdown_connections 2023-11-15T04:28:13.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.459+0000 7f1badffc700 1 -- 172.21.15.47:0/3763545532 wait complete. 2023-11-15T04:28:13.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.460+0000 7f1badffc700 1 Processor -- start 2023-11-15T04:28:13.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.460+0000 7f1badffc700 1 -- start start 2023-11-15T04:28:13.467 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1badffc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:13.467 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1badffc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba810d1e0 con 0x7f1ba80703b0 2023-11-15T04:28:13.467 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1ba77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:13.467 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1ba77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52534/0 (socket says 172.21.15.47:52534) 2023-11-15T04:28:13.467 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1ba77fe700 1 -- 172.21.15.47:0/4275380509 learned_addr learned my addr 172.21.15.47:0/4275380509 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:13.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1ba77fe700 1 -- 172.21.15.47:0/4275380509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9800b200 con 0x7f1ba80703b0 2023-11-15T04:28:13.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1ba77fe700 1 --2- 172.21.15.47:0/4275380509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba810ccf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1b9800e040 tx=0x7f1b98006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:13.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1b8ffff700 1 -- 172.21.15.47:0/4275380509 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9800d9f0 con 0x7f1ba80703b0 2023-11-15T04:28:13.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1b8ffff700 1 -- 172.21.15.47:0/4275380509 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1b9801a9f0 con 0x7f1ba80703b0 2023-11-15T04:28:13.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.461+0000 7f1badffc700 1 -- 172.21.15.47:0/4275380509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba810d3e0 con 0x7f1ba80703b0 2023-11-15T04:28:13.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.462+0000 7f1b8ffff700 1 -- 172.21.15.47:0/4275380509 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9801b8c0 con 0x7f1ba80703b0 2023-11-15T04:28:13.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.462+0000 7f1badffc700 1 -- 172.21.15.47:0/4275380509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba810d7f0 con 0x7f1ba80703b0 2023-11-15T04:28:13.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.463+0000 7f1b8ffff700 1 -- 172.21.15.47:0/4275380509 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1b9800f070 con 0x7f1ba80703b0 2023-11-15T04:28:13.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.463+0000 7f1b8ffff700 1 -- 172.21.15.47:0/4275380509 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1b980257d0 con 0x7f1ba80703b0 2023-11-15T04:28:13.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.463+0000 7f1b8dffb700 1 -- 172.21.15.47:0/4275380509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b80004fa0 con 0x7f1ba80703b0 2023-11-15T04:28:13.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.466+0000 7f1b8ffff700 1 -- 172.21.15.47:0/4275380509 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1b98016070 con 0x7f1ba80703b0 2023-11-15T04:28:13.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:13 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:12.101876+0000 mon.smithi047 (mon.0) 6 : audit [INF] from='client.? 172.21.15.47:0/170803575' entity='client.admin' 2023-11-15T04:28:13.540 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.539+0000 7f1b8dffb700 1 -- 172.21.15.47:0/4275380509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1b80005940 con 0x7f1ba80703b0 2023-11-15T04:28:13.540 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.539+0000 7f1b8ffff700 1 -- 172.21.15.47:0/4275380509 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f1b9801ba20 con 0x7f1ba80703b0 2023-11-15T04:28:13.540 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.542 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2023-11-15T04:28:13.542 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "4f7ad8e8-836f-11ee-8dbe-212e2dc638e7", 2023-11-15T04:28:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:28:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:28:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:28:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:28:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:28:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:28:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2023-11-15T04:28:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:28:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2023-11-15T04:28:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-15T04:28:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:28:13.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:13.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:28:13.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:28:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:28:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:28:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:28:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:28:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:28:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:28:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:28:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:28:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:28:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:28:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:28:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:28:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:28:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:28:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:28:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:28:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:28:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:28:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:28:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:28:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": false, 2023-11-15T04:28:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:28:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:28:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:28:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:28:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:13.558 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:28:13.559 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:13.559 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:28:13.559 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:13.559 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:28:08.486783+0000", 2023-11-15T04:28:13.559 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:28:13.560 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:13.560 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:28:13.560 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2023-11-15T04:28:13.560 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.540+0000 7f1b8dffb700 1 -- 172.21.15.47:0/4275380509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 msgr2=0x7f1ba810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:13.561 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.540+0000 7f1b8dffb700 1 --2- 172.21.15.47:0/4275380509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba810ccf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1b9800e040 tx=0x7f1b98006ee0).stop 2023-11-15T04:28:13.561 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.540+0000 7f1b8dffb700 1 -- 172.21.15.47:0/4275380509 shutdown_connections 2023-11-15T04:28:13.561 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.540+0000 7f1b8dffb700 1 --2- 172.21.15.47:0/4275380509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ba80703b0 0x7f1ba810ccf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:13.561 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.540+0000 7f1b8dffb700 1 -- 172.21.15.47:0/4275380509 >> 172.21.15.47:0/4275380509 conn(0x7f1ba80f63c0 msgr2=0x7f1ba80f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:13.561 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.540+0000 7f1b8dffb700 1 -- 172.21.15.47:0/4275380509 shutdown_connections 2023-11-15T04:28:13.562 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:13.540+0000 7f1b8dffb700 1 -- 172.21.15.47:0/4275380509 wait complete. 2023-11-15T04:28:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:mgr not available, waiting (1/15)... 2023-11-15T04:28:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:13.540480+0000 mon.smithi047 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.47:0/4275380509' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:28:15.898 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:28:16.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.268+0000 7f45800f4700 1 Processor -- start 2023-11-15T04:28:16.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.268+0000 7f45800f4700 1 -- start start 2023-11-15T04:28:16.272 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.268+0000 7f45800f4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f45780fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:16.272 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.268+0000 7f45800f4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45780fb490 con 0x7f45780fab80 2023-11-15T04:28:16.272 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.268+0000 7f457de90700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f45780fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:16.272 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.268+0000 7f457de90700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f45780fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44026/0 (socket says 172.21.15.47:44026) 2023-11-15T04:28:16.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.268+0000 7f457de90700 1 -- 172.21.15.47:0/2321468740 learned_addr learned my addr 172.21.15.47:0/2321468740 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:16.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.269+0000 7f457de90700 1 -- 172.21.15.47:0/2321468740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45780fb5d0 con 0x7f45780fab80 2023-11-15T04:28:16.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.269+0000 7f457de90700 1 --2- 172.21.15.47:0/2321468740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f45780fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f457400b5f0 tx=0x7f4574006ee0).ready entity=mon.0 client_cookie=305cf27b8af3ff1b server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:16.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.269+0000 7f457ce8e700 1 -- 172.21.15.47:0/2321468740 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457400d600 con 0x7f45780fab80 2023-11-15T04:28:16.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.269+0000 7f457ce8e700 1 -- 172.21.15.47:0/2321468740 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f457400dbd0 con 0x7f45780fab80 2023-11-15T04:28:16.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.269+0000 7f457ce8e700 1 -- 172.21.15.47:0/2321468740 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4574011c00 con 0x7f45780fab80 2023-11-15T04:28:16.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.270+0000 7f45800f4700 1 -- 172.21.15.47:0/2321468740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 msgr2=0x7f45780fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:16.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.270+0000 7f45800f4700 1 --2- 172.21.15.47:0/2321468740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f45780fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f457400b5f0 tx=0x7f4574006ee0).stop 2023-11-15T04:28:16.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.270+0000 7f45800f4700 1 -- 172.21.15.47:0/2321468740 shutdown_connections 2023-11-15T04:28:16.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.270+0000 7f45800f4700 1 --2- 172.21.15.47:0/2321468740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f45780fafa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:16.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.270+0000 7f45800f4700 1 -- 172.21.15.47:0/2321468740 >> 172.21.15.47:0/2321468740 conn(0x7f45780f63c0 msgr2=0x7f45780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:16.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.270+0000 7f45800f4700 1 -- 172.21.15.47:0/2321468740 shutdown_connections 2023-11-15T04:28:16.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.270+0000 7f45800f4700 1 -- 172.21.15.47:0/2321468740 wait complete. 2023-11-15T04:28:16.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.271+0000 7f45800f4700 1 Processor -- start 2023-11-15T04:28:16.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.271+0000 7f45800f4700 1 -- start start 2023-11-15T04:28:16.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.271+0000 7f45800f4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f457810d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:16.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.271+0000 7f45800f4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4578070450 con 0x7f45780fab80 2023-11-15T04:28:16.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f457de90700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f457810d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:16.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f457de90700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f457810d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44034/0 (socket says 172.21.15.47:44034) 2023-11-15T04:28:16.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f457de90700 1 -- 172.21.15.47:0/1593834830 learned_addr learned my addr 172.21.15.47:0/1593834830 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:16.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f457de90700 1 -- 172.21.15.47:0/1593834830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457400b200 con 0x7f45780fab80 2023-11-15T04:28:16.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f457de90700 1 --2- 172.21.15.47:0/1593834830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f457810d740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f457400e040 tx=0x7f4574006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:16.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f45667fc700 1 -- 172.21.15.47:0/1593834830 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457401b410 con 0x7f45780fab80 2023-11-15T04:28:16.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f45800f4700 1 -- 172.21.15.47:0/1593834830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4578070650 con 0x7f45780fab80 2023-11-15T04:28:16.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f45667fc700 1 -- 172.21.15.47:0/1593834830 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f457401b9e0 con 0x7f45780fab80 2023-11-15T04:28:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f45667fc700 1 -- 172.21.15.47:0/1593834830 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457401a820 con 0x7f45780fab80 2023-11-15T04:28:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.272+0000 7f45800f4700 1 -- 172.21.15.47:0/1593834830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457810ddf0 con 0x7f45780fab80 2023-11-15T04:28:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.273+0000 7f45667fc700 1 -- 172.21.15.47:0/1593834830 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f457400f070 con 0x7f45780fab80 2023-11-15T04:28:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.273+0000 7f45667fc700 1 -- 172.21.15.47:0/1593834830 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4574025bd0 con 0x7f45780fab80 2023-11-15T04:28:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.273+0000 7f45800f4700 1 -- 172.21.15.47:0/1593834830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4578045ad0 con 0x7f45780fab80 2023-11-15T04:28:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.276+0000 7f45667fc700 1 -- 172.21.15.47:0/1593834830 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4574016070 con 0x7f45780fab80 2023-11-15T04:28:16.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.350+0000 7f45800f4700 1 -- 172.21.15.47:0/1593834830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f45780fc990 con 0x7f45780fab80 2023-11-15T04:28:16.359 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.351+0000 7f45667fc700 1 -- 172.21.15.47:0/1593834830 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f4574025e10 con 0x7f45780fab80 2023-11-15T04:28:16.359 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.359 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2023-11-15T04:28:16.359 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "4f7ad8e8-836f-11ee-8dbe-212e2dc638e7", 2023-11-15T04:28:16.359 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:28:16.360 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:28:16.360 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:28:16.360 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:28:16.360 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:16.360 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:28:16.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:28:16.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2023-11-15T04:28:16.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:16.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:28:16.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2023-11-15T04:28:16.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:16.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-15T04:28:16.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:28:16.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:16.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:28:16.363 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:28:16.363 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:16.363 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:28:16.363 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:16.363 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:28:16.363 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:28:16.364 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:28:16.364 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:28:16.364 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:28:16.364 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:28:16.364 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:16.365 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:28:16.365 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:28:16.365 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:28:16.365 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:28:16.366 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:28:16.366 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:28:16.366 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:28:16.366 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:28:16.366 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:28:16.367 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:16.367 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:28:16.367 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:16.367 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:28:16.367 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:28:16.368 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:16.368 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:28:16.368 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": false, 2023-11-15T04:28:16.368 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:28:16.368 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:28:16.368 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:28:16.369 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:28:16.369 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:16.369 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:28:16.369 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:16.369 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:28:16.370 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:16.370 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:28:08.486783+0000", 2023-11-15T04:28:16.370 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:28:16.370 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:16.370 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:28:16.371 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2023-11-15T04:28:16.371 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.352+0000 7f45800f4700 1 -- 172.21.15.47:0/1593834830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 msgr2=0x7f457810d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:16.371 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.352+0000 7f45800f4700 1 --2- 172.21.15.47:0/1593834830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f457810d740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f457400e040 tx=0x7f4574006ee0).stop 2023-11-15T04:28:16.372 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.352+0000 7f45800f4700 1 -- 172.21.15.47:0/1593834830 shutdown_connections 2023-11-15T04:28:16.372 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.352+0000 7f45800f4700 1 --2- 172.21.15.47:0/1593834830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45780fab80 0x7f457810d740 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:16.372 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.352+0000 7f45800f4700 1 -- 172.21.15.47:0/1593834830 >> 172.21.15.47:0/1593834830 conn(0x7f45780f63c0 msgr2=0x7f45780f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:16.372 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.352+0000 7f45800f4700 1 -- 172.21.15.47:0/1593834830 shutdown_connections 2023-11-15T04:28:16.372 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:16.352+0000 7f45800f4700 1 -- 172.21.15.47:0/1593834830 wait complete. 2023-11-15T04:28:16.647 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:16.352126+0000 mon.smithi047 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.47:0/1593834830' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:28:16.686 INFO:teuthology.orchestra.run.smithi047.stderr:mgr not available, waiting (2/15)... 2023-11-15T04:28:18.688 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:28:19.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:18.660980+0000 mon.smithi047 (mon.0) 9 : cluster [INF] Activating manager daemon smithi047.zgysuz 2023-11-15T04:28:19.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:18.663755+0000 mon.smithi047 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi047.zgysuz(active, starting, since 0.00290025s) 2023-11-15T04:28:19.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.664517+0000 mon.smithi047 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:28:19.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.664747+0000 mon.smithi047 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:28:19.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.664961+0000 mon.smithi047 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:28:19.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.665133+0000 mon.smithi047 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2023-11-15T04:28:19.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.665603+0000 mon.smithi047 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mgr metadata", "who": "smithi047.zgysuz", "id": "smithi047.zgysuz"}]: dispatch 2023-11-15T04:28:19.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:18.667696+0000 mon.smithi047 (mon.0) 16 : cluster [INF] Manager daemon smithi047.zgysuz is now available 2023-11-15T04:28:19.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.675476+0000 mon.smithi047 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:19.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.681472+0000 mon.smithi047 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:28:19.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.682790+0000 mon.smithi047 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:28:19.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.687920+0000 mon.smithi047 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:19.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:18.690384+0000 mon.smithi047 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.47:0/3592456902' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:19.147 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.145+0000 7fbbb806f700 1 Processor -- start 2023-11-15T04:28:19.148 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.146+0000 7fbbb806f700 1 -- start start 2023-11-15T04:28:19.148 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.146+0000 7fbbb806f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:19.148 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.146+0000 7fbbb806f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00fedd0 con 0x7fbbb0070430 2023-11-15T04:28:19.148 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.146+0000 7fbbb5e0b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:19.148 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.146+0000 7fbbb5e0b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44078/0 (socket says 172.21.15.47:44078) 2023-11-15T04:28:19.149 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.146+0000 7fbbb5e0b700 1 -- 172.21.15.47:0/322209516 learned_addr learned my addr 172.21.15.47:0/322209516 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:19.149 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.146+0000 7fbbb5e0b700 1 -- 172.21.15.47:0/322209516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb00fef10 con 0x7fbbb0070430 2023-11-15T04:28:19.149 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.146+0000 7fbbb5e0b700 1 --2- 172.21.15.47:0/322209516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fe8e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbbac00b5f0 tx=0x7fbbac006ee0).ready entity=mon.0 client_cookie=4a14bf86880282f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:19.149 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb4e09700 1 -- 172.21.15.47:0/322209516 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbac00d600 con 0x7fbbb0070430 2023-11-15T04:28:19.149 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb4e09700 1 -- 172.21.15.47:0/322209516 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbbac00dbd0 con 0x7fbbb0070430 2023-11-15T04:28:19.152 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb4e09700 1 -- 172.21.15.47:0/322209516 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbac011c00 con 0x7fbbb0070430 2023-11-15T04:28:19.152 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb806f700 1 -- 172.21.15.47:0/322209516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 msgr2=0x7fbbb00fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:19.152 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb806f700 1 --2- 172.21.15.47:0/322209516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fe8e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbbac00b5f0 tx=0x7fbbac006ee0).stop 2023-11-15T04:28:19.153 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb806f700 1 -- 172.21.15.47:0/322209516 shutdown_connections 2023-11-15T04:28:19.153 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb806f700 1 --2- 172.21.15.47:0/322209516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fe8e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:19.153 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb806f700 1 -- 172.21.15.47:0/322209516 >> 172.21.15.47:0/322209516 conn(0x7fbbb00f63c0 msgr2=0x7fbbb00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:19.153 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb806f700 1 -- 172.21.15.47:0/322209516 shutdown_connections 2023-11-15T04:28:19.153 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.147+0000 7fbbb806f700 1 -- 172.21.15.47:0/322209516 wait complete. 2023-11-15T04:28:19.154 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.148+0000 7fbbb806f700 1 Processor -- start 2023-11-15T04:28:19.154 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.148+0000 7fbbb806f700 1 -- start start 2023-11-15T04:28:19.154 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbbb806f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:19.154 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbbb806f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00fa7e0 con 0x7fbbb0070430 2023-11-15T04:28:19.154 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbbb5e0b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:19.155 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbbb5e0b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44080/0 (socket says 172.21.15.47:44080) 2023-11-15T04:28:19.155 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbbb5e0b700 1 -- 172.21.15.47:0/705698616 learned_addr learned my addr 172.21.15.47:0/705698616 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:19.155 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbbb5e0b700 1 -- 172.21.15.47:0/705698616 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbac00b200 con 0x7fbbb0070430 2023-11-15T04:28:19.155 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbbb5e0b700 1 --2- 172.21.15.47:0/705698616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fc360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbbac00e040 tx=0x7fbbac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:19.155 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbb9e7fc700 1 -- 172.21.15.47:0/705698616 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbac00d9f0 con 0x7fbbb0070430 2023-11-15T04:28:19.156 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbb9e7fc700 1 -- 172.21.15.47:0/705698616 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbbac01a9f0 con 0x7fbbb0070430 2023-11-15T04:28:19.156 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbbb806f700 1 -- 172.21.15.47:0/705698616 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb00fa980 con 0x7fbbb0070430 2023-11-15T04:28:19.156 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbb9e7fc700 1 -- 172.21.15.47:0/705698616 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbac01b8c0 con 0x7fbbb0070430 2023-11-15T04:28:19.156 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.149+0000 7fbbb806f700 1 -- 172.21.15.47:0/705698616 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb00fad90 con 0x7fbbb0070430 2023-11-15T04:28:19.156 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.150+0000 7fbb9e7fc700 1 -- 172.21.15.47:0/705698616 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7fbbac00f070 con 0x7fbbb0070430 2023-11-15T04:28:19.157 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.150+0000 7fbb9e7fc700 1 -- 172.21.15.47:0/705698616 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbbac041120 con 0x7fbbb0070430 2023-11-15T04:28:19.157 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.150+0000 7fbbb806f700 1 -- 172.21.15.47:0/705698616 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb0045ad0 con 0x7fbbb0070430 2023-11-15T04:28:19.157 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.153+0000 7fbb9e7fc700 1 -- 172.21.15.47:0/705698616 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fbbac016070 con 0x7fbbb0070430 2023-11-15T04:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.225+0000 7fbbb806f700 1 -- 172.21.15.47:0/705698616 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbbb01047f0 con 0x7fbbb0070430 2023-11-15T04:28:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.225+0000 7fbb9e7fc700 1 -- 172.21.15.47:0/705698616 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fbbac01ba20 con 0x7fbbb0070430 2023-11-15T04:28:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2023-11-15T04:28:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "4f7ad8e8-836f-11ee-8dbe-212e2dc638e7", 2023-11-15T04:28:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:28:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:28:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:28:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:28:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:28:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:28:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2023-11-15T04:28:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:28:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2023-11-15T04:28:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-15T04:28:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:28:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:28:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:28:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:28:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:28:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:28:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:28:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:28:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:28:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:28:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:28:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:28:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:28:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:28:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:28:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:28:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:28:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:28:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:28:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:28:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:28:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:28:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:28:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": false, 2023-11-15T04:28:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:28:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:28:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:28:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:28:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:28:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:28:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:28:08.486783+0000", 2023-11-15T04:28:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:28:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:28:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2023-11-15T04:28:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.226+0000 7fbbb806f700 1 -- 172.21.15.47:0/705698616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 msgr2=0x7fbbb00fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.226+0000 7fbbb806f700 1 --2- 172.21.15.47:0/705698616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fc360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbbac00e040 tx=0x7fbbac006ee0).stop 2023-11-15T04:28:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.226+0000 7fbbb806f700 1 -- 172.21.15.47:0/705698616 shutdown_connections 2023-11-15T04:28:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.226+0000 7fbbb806f700 1 --2- 172.21.15.47:0/705698616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbb0070430 0x7fbbb00fc360 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.226+0000 7fbbb806f700 1 -- 172.21.15.47:0/705698616 >> 172.21.15.47:0/705698616 conn(0x7fbbb00f63c0 msgr2=0x7fbbb00f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:19.245 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.226+0000 7fbbb806f700 1 -- 172.21.15.47:0/705698616 shutdown_connections 2023-11-15T04:28:19.245 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:19.226+0000 7fbbb806f700 1 -- 172.21.15.47:0/705698616 wait complete. 2023-11-15T04:28:19.579 INFO:teuthology.orchestra.run.smithi047.stderr:mgr not available, waiting (3/15)... 2023-11-15T04:28:20.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:19 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:19.226459+0000 mon.smithi047 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.47:0/705698616' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:28:20.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:19 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:19.666936+0000 mon.smithi047 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi047.zgysuz(active, since 1.00608s) 2023-11-15T04:28:21.581 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:28:21.960 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.957+0000 7f4b97f97700 1 Processor -- start 2023-11-15T04:28:21.960 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.958+0000 7f4b97f97700 1 -- start start 2023-11-15T04:28:21.960 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.958+0000 7f4b97f97700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b90100d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:21.960 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.958+0000 7f4b97f97700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900fc760 con 0x7f4b90070420 2023-11-15T04:28:21.961 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.958+0000 7f4b95d33700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b90100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:21.961 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.958+0000 7f4b95d33700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b90100d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44084/0 (socket says 172.21.15.47:44084) 2023-11-15T04:28:21.961 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.958+0000 7f4b95d33700 1 -- 172.21.15.47:0/4024061391 learned_addr learned my addr 172.21.15.47:0/4024061391 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:21.961 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.958+0000 7f4b95d33700 1 -- 172.21.15.47:0/4024061391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b90101260 con 0x7f4b90070420 2023-11-15T04:28:21.962 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.958+0000 7f4b95d33700 1 --2- 172.21.15.47:0/4024061391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b90100d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4b8c00b5f0 tx=0x7f4b8c006ee0).ready entity=mon.0 client_cookie=4d84435116af3e86 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:21.962 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.959+0000 7f4b94d31700 1 -- 172.21.15.47:0/4024061391 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b8c00d600 con 0x7f4b90070420 2023-11-15T04:28:21.962 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.959+0000 7f4b94d31700 1 -- 172.21.15.47:0/4024061391 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4b8c00dbd0 con 0x7f4b90070420 2023-11-15T04:28:21.962 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.959+0000 7f4b94d31700 1 -- 172.21.15.47:0/4024061391 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b8c011c00 con 0x7f4b90070420 2023-11-15T04:28:21.962 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.959+0000 7f4b97f97700 1 -- 172.21.15.47:0/4024061391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 msgr2=0x7f4b90100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:21.963 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.959+0000 7f4b97f97700 1 --2- 172.21.15.47:0/4024061391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b90100d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4b8c00b5f0 tx=0x7f4b8c006ee0).stop 2023-11-15T04:28:21.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.959+0000 7f4b97f97700 1 -- 172.21.15.47:0/4024061391 shutdown_connections 2023-11-15T04:28:21.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.959+0000 7f4b97f97700 1 --2- 172.21.15.47:0/4024061391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b90100d70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:21.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.959+0000 7f4b97f97700 1 -- 172.21.15.47:0/4024061391 >> 172.21.15.47:0/4024061391 conn(0x7f4b900f63b0 msgr2=0x7f4b900f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:21.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.959+0000 7f4b97f97700 1 -- 172.21.15.47:0/4024061391 shutdown_connections 2023-11-15T04:28:21.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.960+0000 7f4b97f97700 1 -- 172.21.15.47:0/4024061391 wait complete. 2023-11-15T04:28:21.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.960+0000 7f4b97f97700 1 Processor -- start 2023-11-15T04:28:21.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.960+0000 7f4b97f97700 1 -- start start 2023-11-15T04:28:21.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b97f97700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b9010ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:21.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b97f97700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9010d1c0 con 0x7f4b90070420 2023-11-15T04:28:21.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b95d33700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b9010ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:21.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b95d33700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b9010ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44092/0 (socket says 172.21.15.47:44092) 2023-11-15T04:28:21.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b95d33700 1 -- 172.21.15.47:0/1266824932 learned_addr learned my addr 172.21.15.47:0/1266824932 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:21.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b95d33700 1 -- 172.21.15.47:0/1266824932 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c00b200 con 0x7f4b90070420 2023-11-15T04:28:21.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b95d33700 1 --2- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b9010ccd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f4b8c00e040 tx=0x7f4b8c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:21.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b7e7fc700 1 -- 172.21.15.47:0/1266824932 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b8c00d9f0 con 0x7f4b90070420 2023-11-15T04:28:21.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b7e7fc700 1 -- 172.21.15.47:0/1266824932 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4b8c01a9f0 con 0x7f4b90070420 2023-11-15T04:28:21.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9010d3c0 con 0x7f4b90070420 2023-11-15T04:28:21.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b7e7fc700 1 -- 172.21.15.47:0/1266824932 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b8c01b8c0 con 0x7f4b90070420 2023-11-15T04:28:21.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.961+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9010d7d0 con 0x7f4b90070420 2023-11-15T04:28:21.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.962+0000 7f4b7e7fc700 1 -- 172.21.15.47:0/1266824932 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f4b8c00f070 con 0x7f4b90070420 2023-11-15T04:28:21.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.962+0000 7f4b7e7fc700 1 --2- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4b7802fd00 0x7f4b780321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:21.970 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.962+0000 7f4b7e7fc700 1 -- 172.21.15.47:0/1266824932 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4b8c041220 con 0x7f4b90070420 2023-11-15T04:28:21.970 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.963+0000 7f4b95532700 1 --2- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4b7802fd00 0x7f4b780321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:21.970 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.963+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b90101dc0 con 0x7f4b90070420 2023-11-15T04:28:21.970 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.963+0000 7f4b95532700 1 --2- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4b7802fd00 0x7f4b780321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f4b800096a0 tx=0x7f4b80006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:21.971 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:21.968+0000 7f4b7e7fc700 1 -- 172.21.15.47:0/1266824932 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b8c016070 con 0x7f4b90070420 2023-11-15T04:28:22.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:21 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:20.715539+0000 mon.smithi047 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi047.zgysuz(active, since 2s) 2023-11-15T04:28:22.203 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.195+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4b9010e600 con 0x7f4b90070420 2023-11-15T04:28:22.203 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.195+0000 7f4b7e7fc700 1 -- 172.21.15.47:0/1266824932 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f4b8c006a70 con 0x7f4b90070420 2023-11-15T04:28:22.203 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.203 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2023-11-15T04:28:22.204 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "4f7ad8e8-836f-11ee-8dbe-212e2dc638e7", 2023-11-15T04:28:22.204 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:28:22.204 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:28:22.204 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:28:22.204 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:28:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:28:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:28:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2023-11-15T04:28:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:22.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:28:22.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2023-11-15T04:28:22.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:22.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-11-15T04:28:22.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:28:22.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:22.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:28:22.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:28:22.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:22.208 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:28:22.208 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:22.208 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:28:22.208 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:28:22.208 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:28:22.208 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:28:22.209 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:28:22.209 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:28:22.209 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:22.209 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:28:22.209 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:28:22.210 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:28:22.210 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:28:22.210 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:28:22.210 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:28:22.210 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:28:22.211 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:28:22.211 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:28:22.211 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:22.211 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:28:22.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:22.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:28:22.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:28:22.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:22.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:28:22.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": true, 2023-11-15T04:28:22.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:28:22.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:28:22.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:28:22.214 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:28:22.214 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2023-11-15T04:28:22.214 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:28:22.214 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:22.214 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:28:22.215 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:28:22.215 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:28:08.486783+0000", 2023-11-15T04:28:22.215 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:28:22.215 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2023-11-15T04:28:22.215 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:28:22.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2023-11-15T04:28:22.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4b7802fd00 msgr2=0x7f4b780321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:22.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 --2- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4b7802fd00 0x7f4b780321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f4b800096a0 tx=0x7f4b80006b40).stop 2023-11-15T04:28:22.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 msgr2=0x7f4b9010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:22.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 --2- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b9010ccd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f4b8c00e040 tx=0x7f4b8c006ee0).stop 2023-11-15T04:28:22.217 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 shutdown_connections 2023-11-15T04:28:22.217 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 --2- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4b7802fd00 0x7f4b780321c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:22.217 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 --2- 172.21.15.47:0/1266824932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b90070420 0x7f4b9010ccd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:22.217 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 >> 172.21.15.47:0/1266824932 conn(0x7f4b900f63b0 msgr2=0x7f4b900f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:22.217 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 shutdown_connections 2023-11-15T04:28:22.218 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.197+0000 7f4b97f97700 1 -- 172.21.15.47:0/1266824932 wait complete. 2023-11-15T04:28:22.536 INFO:teuthology.orchestra.run.smithi047.stderr:mgr is available 2023-11-15T04:28:22.537 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling cephadm module... 2023-11-15T04:28:22.537 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-15T04:28:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.918+0000 7eff3759e700 1 Processor -- start 2023-11-15T04:28:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.918+0000 7eff3759e700 1 -- start start 2023-11-15T04:28:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.918+0000 7eff3759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff380fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.918+0000 7eff3759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380fec60 con 0x7eff380703f0 2023-11-15T04:28:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.918+0000 7eff3659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff380fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.918+0000 7eff3659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff380fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44106/0 (socket says 172.21.15.47:44106) 2023-11-15T04:28:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.918+0000 7eff3659c700 1 -- 172.21.15.47:0/3463933735 learned_addr learned my addr 172.21.15.47:0/3463933735 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.919+0000 7eff3659c700 1 -- 172.21.15.47:0/3463933735 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff380feda0 con 0x7eff380703f0 2023-11-15T04:28:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.919+0000 7eff3659c700 1 --2- 172.21.15.47:0/3463933735 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff380fe770 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7eff2800b5f0 tx=0x7eff28006ee0).ready entity=mon.0 client_cookie=a9ef0d58f2f1223a server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.919+0000 7eff35d9b700 1 -- 172.21.15.47:0/3463933735 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2800d600 con 0x7eff380703f0 2023-11-15T04:28:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.919+0000 7eff35d9b700 1 -- 172.21.15.47:0/3463933735 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7eff2800dbd0 con 0x7eff380703f0 2023-11-15T04:28:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.919+0000 7eff35d9b700 1 -- 172.21.15.47:0/3463933735 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff28011c00 con 0x7eff380703f0 2023-11-15T04:28:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.920+0000 7eff3759e700 1 -- 172.21.15.47:0/3463933735 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 msgr2=0x7eff380fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.920+0000 7eff3759e700 1 --2- 172.21.15.47:0/3463933735 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff380fe770 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7eff2800b5f0 tx=0x7eff28006ee0).stop 2023-11-15T04:28:22.925 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.920+0000 7eff3759e700 1 -- 172.21.15.47:0/3463933735 shutdown_connections 2023-11-15T04:28:22.926 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.920+0000 7eff3759e700 1 --2- 172.21.15.47:0/3463933735 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff380fe770 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:22.926 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.920+0000 7eff3759e700 1 -- 172.21.15.47:0/3463933735 >> 172.21.15.47:0/3463933735 conn(0x7eff380f6210 msgr2=0x7eff380f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:22.926 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.920+0000 7eff3759e700 1 -- 172.21.15.47:0/3463933735 shutdown_connections 2023-11-15T04:28:22.926 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.920+0000 7eff3759e700 1 -- 172.21.15.47:0/3463933735 wait complete. 2023-11-15T04:28:22.926 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.920+0000 7eff3759e700 1 Processor -- start 2023-11-15T04:28:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.921+0000 7eff3759e700 1 -- start start 2023-11-15T04:28:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.921+0000 7eff3759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff3810a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.921+0000 7eff3759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3810ad00 con 0x7eff380703f0 2023-11-15T04:28:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.921+0000 7eff3659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff3810a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.921+0000 7eff3659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff3810a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44120/0 (socket says 172.21.15.47:44120) 2023-11-15T04:28:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.921+0000 7eff3659c700 1 -- 172.21.15.47:0/1534006523 learned_addr learned my addr 172.21.15.47:0/1534006523 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.921+0000 7eff3659c700 1 -- 172.21.15.47:0/1534006523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2800b200 con 0x7eff380703f0 2023-11-15T04:28:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.921+0000 7eff3659c700 1 --2- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff3810a810 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7eff2800b190 tx=0x7eff28006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.922+0000 7eff2effd700 1 -- 172.21.15.47:0/1534006523 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2801b410 con 0x7eff380703f0 2023-11-15T04:28:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.922+0000 7eff2effd700 1 -- 172.21.15.47:0/1534006523 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7eff2801b9e0 con 0x7eff380703f0 2023-11-15T04:28:22.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.922+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3810af00 con 0x7eff380703f0 2023-11-15T04:28:22.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.922+0000 7eff2effd700 1 -- 172.21.15.47:0/1534006523 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2801a880 con 0x7eff380703f0 2023-11-15T04:28:22.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.922+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3810b310 con 0x7eff380703f0 2023-11-15T04:28:22.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.922+0000 7eff2effd700 1 -- 172.21.15.47:0/1534006523 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7eff2800f070 con 0x7eff380703f0 2023-11-15T04:28:22.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.923+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff38045ad0 con 0x7eff380703f0 2023-11-15T04:28:22.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.923+0000 7eff2effd700 1 --2- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7eff1802fd50 0x7eff18032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:22.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.923+0000 7eff2effd700 1 -- 172.21.15.47:0/1534006523 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7eff280411a0 con 0x7eff380703f0 2023-11-15T04:28:22.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.923+0000 7eff2ffff700 1 --2- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7eff1802fd50 0x7eff18032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:22.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.923+0000 7eff2ffff700 1 --2- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7eff1802fd50 0x7eff18032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7eff200096a0 tx=0x7eff20006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:22.931 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:22.927+0000 7eff2effd700 1 -- 172.21.15.47:0/1534006523 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff280224f0 con 0x7eff380703f0 2023-11-15T04:28:23.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:22.196599+0000 mon.smithi047 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.47:0/1266824932' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:28:23.133 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.132+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7eff380ffa20 con 0x7eff380703f0 2023-11-15T04:28:23.718 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.717+0000 7eff2effd700 1 -- 172.21.15.47:0/1534006523 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7eff28025460 con 0x7eff380703f0 2023-11-15T04:28:23.719 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.719+0000 7eff2effd700 1 -- 172.21.15.47:0/1534006523 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7eff28043c20 con 0x7eff380703f0 2023-11-15T04:28:23.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7eff1802fd50 msgr2=0x7eff18032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:23.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 --2- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7eff1802fd50 0x7eff18032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7eff200096a0 tx=0x7eff20006b40).stop 2023-11-15T04:28:23.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 msgr2=0x7eff3810a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:23.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 --2- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff3810a810 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7eff2800b190 tx=0x7eff28006ee0).stop 2023-11-15T04:28:23.724 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 shutdown_connections 2023-11-15T04:28:23.724 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 --2- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7eff1802fd50 0x7eff18032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:23.724 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 --2- 172.21.15.47:0/1534006523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff380703f0 0x7eff3810a810 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:23.724 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 >> 172.21.15.47:0/1534006523 conn(0x7eff380f6210 msgr2=0x7eff380f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:23.724 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 shutdown_connections 2023-11-15T04:28:23.725 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:23.721+0000 7eff3759e700 1 -- 172.21.15.47:0/1534006523 wait complete. 2023-11-15T04:28:24.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:23 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:23.133972+0000 mon.smithi047 (mon.0) 26 : audit [INF] from='client.? 172.21.15.47:0/1534006523' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-15T04:28:24.062 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T04:28:24.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.425+0000 7f4de75ad700 1 Processor -- start 2023-11-15T04:28:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.425+0000 7f4de75ad700 1 -- start start 2023-11-15T04:28:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.425+0000 7f4de75ad700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de0100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.425+0000 7f4de75ad700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de00fc750 con 0x7f4de00703b0 2023-11-15T04:28:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.426+0000 7f4de5349700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de0100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.426+0000 7f4de5349700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de0100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56796/0 (socket says 172.21.15.47:56796) 2023-11-15T04:28:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.426+0000 7f4de5349700 1 -- 172.21.15.47:0/3951614746 learned_addr learned my addr 172.21.15.47:0/3951614746 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.426+0000 7f4de5349700 1 -- 172.21.15.47:0/3951614746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de0101250 con 0x7f4de00703b0 2023-11-15T04:28:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.426+0000 7f4de5349700 1 --2- 172.21.15.47:0/3951614746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de0100d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4ddc00b5f0 tx=0x7f4ddc006ee0).ready entity=mon.0 client_cookie=49be260a1e6f6f21 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.426+0000 7f4dcffff700 1 -- 172.21.15.47:0/3951614746 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ddc00d600 con 0x7f4de00703b0 2023-11-15T04:28:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.426+0000 7f4dcffff700 1 -- 172.21.15.47:0/3951614746 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4ddc00dbd0 con 0x7f4de00703b0 2023-11-15T04:28:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.426+0000 7f4dcffff700 1 -- 172.21.15.47:0/3951614746 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ddc011c00 con 0x7f4de00703b0 2023-11-15T04:28:24.432 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.427+0000 7f4de75ad700 1 -- 172.21.15.47:0/3951614746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 msgr2=0x7f4de0100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:24.432 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.427+0000 7f4de75ad700 1 --2- 172.21.15.47:0/3951614746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de0100d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4ddc00b5f0 tx=0x7f4ddc006ee0).stop 2023-11-15T04:28:24.433 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.427+0000 7f4de75ad700 1 -- 172.21.15.47:0/3951614746 shutdown_connections 2023-11-15T04:28:24.433 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.427+0000 7f4de75ad700 1 --2- 172.21.15.47:0/3951614746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de0100d60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:24.433 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.427+0000 7f4de75ad700 1 -- 172.21.15.47:0/3951614746 >> 172.21.15.47:0/3951614746 conn(0x7f4de00f63c0 msgr2=0x7f4de00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:24.433 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.427+0000 7f4de75ad700 1 -- 172.21.15.47:0/3951614746 shutdown_connections 2023-11-15T04:28:24.433 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.427+0000 7f4de75ad700 1 -- 172.21.15.47:0/3951614746 wait complete. 2023-11-15T04:28:24.434 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.428+0000 7f4de75ad700 1 Processor -- start 2023-11-15T04:28:24.434 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.428+0000 7f4de75ad700 1 -- start start 2023-11-15T04:28:24.434 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.428+0000 7f4de75ad700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:24.434 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.428+0000 7f4de75ad700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de010d100 con 0x7f4de00703b0 2023-11-15T04:28:24.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.428+0000 7f4de5349700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:24.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.428+0000 7f4de5349700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56800/0 (socket says 172.21.15.47:56800) 2023-11-15T04:28:24.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.428+0000 7f4de5349700 1 -- 172.21.15.47:0/16953438 learned_addr learned my addr 172.21.15.47:0/16953438 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:24.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.429+0000 7f4de5349700 1 -- 172.21.15.47:0/16953438 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc00b200 con 0x7f4de00703b0 2023-11-15T04:28:24.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.429+0000 7f4de5349700 1 --2- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de010cc10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4ddc00b190 tx=0x7f4ddc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:24.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.429+0000 7f4dcdffb700 1 -- 172.21.15.47:0/16953438 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ddc01b410 con 0x7f4de00703b0 2023-11-15T04:28:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.429+0000 7f4dcdffb700 1 -- 172.21.15.47:0/16953438 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4ddc01b9e0 con 0x7f4de00703b0 2023-11-15T04:28:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.429+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de010d300 con 0x7f4de00703b0 2023-11-15T04:28:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.429+0000 7f4dcdffb700 1 -- 172.21.15.47:0/16953438 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ddc01a880 con 0x7f4de00703b0 2023-11-15T04:28:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.429+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de010d710 con 0x7f4de00703b0 2023-11-15T04:28:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.430+0000 7f4dcdffb700 1 -- 172.21.15.47:0/16953438 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4ddc00f070 con 0x7f4de00703b0 2023-11-15T04:28:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.430+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de0045ad0 con 0x7f4de00703b0 2023-11-15T04:28:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.431+0000 7f4dcdffb700 1 --2- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4dc802fdb0 0x7f4dc8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.431+0000 7f4dcdffb700 1 -- 172.21.15.47:0/16953438 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4ddc041160 con 0x7f4de00703b0 2023-11-15T04:28:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.431+0000 7f4de4b48700 1 -- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4dc802fdb0 msgr2=0x7f4dc8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/656622912 2023-11-15T04:28:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.431+0000 7f4de4b48700 1 --2- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4dc802fdb0 0x7f4dc8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:28:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.435+0000 7f4dcdffb700 1 -- 172.21.15.47:0/16953438 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ddc0257e0 con 0x7f4de00703b0 2023-11-15T04:28:24.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.632+0000 7f4de4b48700 1 -- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4dc802fdb0 msgr2=0x7f4dc8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/656622912 2023-11-15T04:28:24.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.632+0000 7f4de4b48700 1 --2- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4dc802fdb0 0x7f4dc8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:28:24.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.684+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f4de010e820 con 0x7f4de00703b0 2023-11-15T04:28:24.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.685+0000 7f4dcdffb700 1 -- 172.21.15.47:0/16953438 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f4ddc016070 con 0x7f4de00703b0 2023-11-15T04:28:24.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2023-11-15T04:28:24.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-15T04:28:24.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": true, 2023-11-15T04:28:24.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "active_name": "smithi047.zgysuz", 2023-11-15T04:28:24.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T04:28:24.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2023-11-15T04:28:24.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.686+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4dc802fdb0 msgr2=0x7f4dc8032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:28:24.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.686+0000 7f4de75ad700 1 --2- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4dc802fdb0 0x7f4dc8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:28:24.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.687+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 msgr2=0x7f4de010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:24.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.687+0000 7f4de75ad700 1 --2- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de010cc10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4ddc00b190 tx=0x7f4ddc006ee0).stop 2023-11-15T04:28:24.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.687+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 shutdown_connections 2023-11-15T04:28:24.689 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.687+0000 7f4de75ad700 1 --2- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f4dc802fdb0 0x7f4dc8032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:28:24.689 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.687+0000 7f4de75ad700 1 --2- 172.21.15.47:0/16953438 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4de00703b0 0x7f4de010cc10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:24.689 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.687+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 >> 172.21.15.47:0/16953438 conn(0x7f4de00f63c0 msgr2=0x7f4de00f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:24.689 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.687+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 shutdown_connections 2023-11-15T04:28:24.689 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:24.687+0000 7f4de75ad700 1 -- 172.21.15.47:0/16953438 wait complete. 2023-11-15T04:28:24.971 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:23.720127+0000 mon.smithi047 (mon.0) 27 : audit [INF] from='client.? 172.21.15.47:0/1534006523' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-15T04:28:24.972 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:23.720189+0000 mon.smithi047 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi047.zgysuz(active, since 5s) 2023-11-15T04:28:24.972 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:24.684954+0000 mon.smithi047 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.47:0/16953438' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T04:28:25.028 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for the mgr to restart... 2023-11-15T04:28:25.028 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr epoch 5... 2023-11-15T04:28:25.028 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T04:28:25.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.427+0000 7f7c6db1a700 1 Processor -- start 2023-11-15T04:28:25.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.427+0000 7f7c6db1a700 1 -- start start 2023-11-15T04:28:25.429 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.427+0000 7f7c6db1a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c680fafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:25.432 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.427+0000 7f7c6db1a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680fb4b0 con 0x7f7c680faba0 2023-11-15T04:28:25.433 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.427+0000 7f7c677fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c680fafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:25.433 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.427+0000 7f7c677fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c680fafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56816/0 (socket says 172.21.15.47:56816) 2023-11-15T04:28:25.433 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.427+0000 7f7c677fe700 1 -- 172.21.15.47:0/292573150 learned_addr learned my addr 172.21.15.47:0/292573150 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:25.433 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.428+0000 7f7c677fe700 1 -- 172.21.15.47:0/292573150 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c680fb5f0 con 0x7f7c680faba0 2023-11-15T04:28:25.434 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.428+0000 7f7c677fe700 1 --2- 172.21.15.47:0/292573150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c680fafc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f7c5800b5f0 tx=0x7f7c58006ee0).ready entity=mon.0 client_cookie=dea4db03b7bdd006 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:25.434 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.428+0000 7f7c667fc700 1 -- 172.21.15.47:0/292573150 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c5800d600 con 0x7f7c680faba0 2023-11-15T04:28:25.434 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.428+0000 7f7c667fc700 1 -- 172.21.15.47:0/292573150 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7c5800dbd0 con 0x7f7c680faba0 2023-11-15T04:28:25.434 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.428+0000 7f7c667fc700 1 -- 172.21.15.47:0/292573150 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c58011c00 con 0x7f7c680faba0 2023-11-15T04:28:25.434 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.429+0000 7f7c6db1a700 1 -- 172.21.15.47:0/292573150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 msgr2=0x7f7c680fafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:25.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.429+0000 7f7c6db1a700 1 --2- 172.21.15.47:0/292573150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c680fafc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f7c5800b5f0 tx=0x7f7c58006ee0).stop 2023-11-15T04:28:25.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.429+0000 7f7c6db1a700 1 -- 172.21.15.47:0/292573150 shutdown_connections 2023-11-15T04:28:25.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.429+0000 7f7c6db1a700 1 --2- 172.21.15.47:0/292573150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c680fafc0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:25.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.429+0000 7f7c6db1a700 1 -- 172.21.15.47:0/292573150 >> 172.21.15.47:0/292573150 conn(0x7f7c680f63c0 msgr2=0x7f7c680f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:25.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.429+0000 7f7c6db1a700 1 -- 172.21.15.47:0/292573150 shutdown_connections 2023-11-15T04:28:25.436 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.429+0000 7f7c6db1a700 1 -- 172.21.15.47:0/292573150 wait complete. 2023-11-15T04:28:25.436 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.430+0000 7f7c6db1a700 1 Processor -- start 2023-11-15T04:28:25.436 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.430+0000 7f7c6db1a700 1 -- start start 2023-11-15T04:28:25.436 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.430+0000 7f7c6db1a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c6810d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:25.436 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.430+0000 7f7c6db1a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68070450 con 0x7f7c680faba0 2023-11-15T04:28:25.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c677fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c6810d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:25.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c677fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c6810d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56826/0 (socket says 172.21.15.47:56826) 2023-11-15T04:28:25.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c677fe700 1 -- 172.21.15.47:0/4208439258 learned_addr learned my addr 172.21.15.47:0/4208439258 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:25.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c677fe700 1 -- 172.21.15.47:0/4208439258 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5800b200 con 0x7f7c680faba0 2023-11-15T04:28:25.437 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c677fe700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c6810d720 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f7c5800b1d0 tx=0x7f7c58006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:25.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c5801b410 con 0x7f7c680faba0 2023-11-15T04:28:25.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7c5801b9e0 con 0x7f7c680faba0 2023-11-15T04:28:25.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c68070650 con 0x7f7c680faba0 2023-11-15T04:28:25.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c5801a840 con 0x7f7c680faba0 2023-11-15T04:28:25.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.431+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6810ddd0 con 0x7f7c680faba0 2023-11-15T04:28:25.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.432+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7c5800f070 con 0x7f7c680faba0 2023-11-15T04:28:25.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.432+0000 7f7c4ffff700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 0x7f7c54032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:25.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.432+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 --> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c68059100 con 0x7f7c5402fd50 2023-11-15T04:28:25.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.432+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7c580420d0 con 0x7f7c680faba0 2023-11-15T04:28:25.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.432+0000 7f7c66ffd700 1 -- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 msgr2=0x7f7c54032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/656622912 2023-11-15T04:28:25.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.433+0000 7f7c66ffd700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 0x7f7c54032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:28:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.634+0000 7f7c66ffd700 1 -- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 msgr2=0x7f7c54032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/656622912 2023-11-15T04:28:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:25.634+0000 7f7c66ffd700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 0x7f7c54032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:28:26.036 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:26.035+0000 7f7c66ffd700 1 -- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 msgr2=0x7f7c54032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/656622912 2023-11-15T04:28:26.036 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:26.035+0000 7f7c66ffd700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 0x7f7c54032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T04:28:26.837 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:26.836+0000 7f7c66ffd700 1 -- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 msgr2=0x7f7c54032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/656622912 2023-11-15T04:28:26.837 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:26.836+0000 7f7c66ffd700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 0x7f7c54032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T04:28:28.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:28.437+0000 7f7c66ffd700 1 -- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 msgr2=0x7f7c54032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/656622912 2023-11-15T04:28:28.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:28.437+0000 7f7c66ffd700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 0x7f7c54032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T04:28:29.450 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:29.449+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f7c580257e0 con 0x7f7c680faba0 2023-11-15T04:28:29.450 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:29.449+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 msgr2=0x7f7c54032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:28:29.451 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:29.449+0000 7f7c4ffff700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 0x7f7c54032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:28:29.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:29 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:29.397995+0000 mon.smithi047 (mon.0) 30 : cluster [INF] Active manager daemon smithi047.zgysuz restarted 2023-11-15T04:28:29.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:29 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:29.398562+0000 mon.smithi047 (mon.0) 31 : cluster [INF] Activating manager daemon smithi047.zgysuz 2023-11-15T04:28:29.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:29 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:29.400537+0000 mon.smithi047 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-15T04:28:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.457+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f7c5801b740 con 0x7f7c680faba0 2023-11-15T04:28:30.459 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.457+0000 7f7c4ffff700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7c54033670 0x7f7c54035a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:30.459 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.457+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c68059100 con 0x7f7c54033670 2023-11-15T04:28:30.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.458+0000 7f7c66ffd700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7c54033670 0x7f7c54035a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:30.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.459+0000 7f7c66ffd700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7c54033670 0x7f7c54035a60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7c5c006fd0 tx=0x7f7c5c006cf0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:30.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.459+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 <== mgr.14116 v2:172.21.15.47:6800/1915104696 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f7c68059100 con 0x7f7c54033670 2023-11-15T04:28:30.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.466+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f7c68045ad0 con 0x7f7c54033670 2023-11-15T04:28:30.467 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.466+0000 7f7c4ffff700 1 -- 172.21.15.47:0/4208439258 <== mgr.14116 v2:172.21.15.47:6800/1915104696 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f7c68045ad0 con 0x7f7c54033670 2023-11-15T04:28:30.467 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2023-11-15T04:28:30.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-15T04:28:30.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T04:28:30.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2023-11-15T04:28:30.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.466+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7c54033670 msgr2=0x7f7c54035a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:30.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.466+0000 7f7c6db1a700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7c54033670 0x7f7c54035a60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7c5c006fd0 tx=0x7f7c5c006cf0).stop 2023-11-15T04:28:30.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.466+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 msgr2=0x7f7c6810d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:30.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.466+0000 7f7c6db1a700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c6810d720 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f7c5800b1d0 tx=0x7f7c58006ee0).stop 2023-11-15T04:28:30.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.467+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 shutdown_connections 2023-11-15T04:28:30.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.467+0000 7f7c6db1a700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7c54033670 0x7f7c54035a60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:30.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.467+0000 7f7c6db1a700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c680faba0 0x7f7c6810d720 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:30.471 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.467+0000 7f7c6db1a700 1 --2- 172.21.15.47:0/4208439258 >> [v2:172.21.15.47:6800/656622912,v1:172.21.15.47:6801/656622912] conn(0x7f7c5402fd50 0x7f7c54032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:28:30.471 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.467+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 >> 172.21.15.47:0/4208439258 conn(0x7f7c680f63c0 msgr2=0x7f7c680f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:30.471 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.467+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 shutdown_connections 2023-11-15T04:28:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:30.467+0000 7f7c6db1a700 1 -- 172.21.15.47:0/4208439258 wait complete. 2023-11-15T04:28:30.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:29.451665+0000 mon.smithi047 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi047.zgysuz(active, starting, since 0.0532137s) 2023-11-15T04:28:30.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.454524+0000 mon.smithi047 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2023-11-15T04:28:30.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.455408+0000 mon.smithi047 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mgr metadata", "who": "smithi047.zgysuz", "id": "smithi047.zgysuz"}]: dispatch 2023-11-15T04:28:30.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.456247+0000 mon.smithi047 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:28:30.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.456397+0000 mon.smithi047 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:28:30.759 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.456539+0000 mon.smithi047 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:28:30.759 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:29.470754+0000 mon.smithi047 (mon.0) 39 : cluster [INF] Manager daemon smithi047.zgysuz is now available 2023-11-15T04:28:30.759 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.486660+0000 mon.smithi047 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:30.759 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.489714+0000 mon.smithi047 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:30.759 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.492489+0000 mon.smithi047 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:30.760 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.594811+0000 mon.smithi047 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:30.760 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.595649+0000 mon.smithi047 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:28:30.760 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.597028+0000 mon.smithi047 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:28:30.760 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.598624+0000 mon.smithi047 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:28:30.760 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.609784+0000 mon.smithi047 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:28:30.760 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:29.611141+0000 mon.smithi047 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:28:31.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:31 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:30.460274+0000 mgr.smithi047.zgysuz (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T04:28:31.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:31 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:30.461120+0000 mon.smithi047 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi047.zgysuz(active, since 1.06266s) 2023-11-15T04:28:31.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:31 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:30.467448+0000 mgr.smithi047.zgysuz (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T04:28:32.445 INFO:teuthology.orchestra.run.smithi047.stderr:mgr epoch 5 is available 2023-11-15T04:28:32.445 INFO:teuthology.orchestra.run.smithi047.stderr:Setting orchestrator backend to cephadm... 2023-11-15T04:28:32.445 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-15T04:28:32.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:31.463170+0000 mon.smithi047 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi047.zgysuz(active, since 2s) 2023-11-15T04:28:33.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.710+0000 7fddfd79c700 1 Processor -- start 2023-11-15T04:28:33.903 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.710+0000 7fddfd79c700 1 -- start start 2023-11-15T04:28:33.903 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.710+0000 7fddfd79c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf8070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:33.903 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.710+0000 7fddfd79c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf8072940 con 0x7fddf8072520 2023-11-15T04:28:33.904 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.710+0000 7fddf6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf8070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:33.904 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.710+0000 7fddf6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf8070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56890/0 (socket says 172.21.15.47:56890) 2023-11-15T04:28:33.904 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.710+0000 7fddf6ffd700 1 -- 172.21.15.47:0/296423084 learned_addr learned my addr 172.21.15.47:0/296423084 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:33.904 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.711+0000 7fddf6ffd700 1 -- 172.21.15.47:0/296423084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8070fb0 con 0x7fddf8072520 2023-11-15T04:28:33.904 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.711+0000 7fddf6ffd700 1 --2- 172.21.15.47:0/296423084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf8070ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fdde000b5f0 tx=0x7fdde0006ee0).ready entity=mon.0 client_cookie=b5db9a52f175dcb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:33.905 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.711+0000 7fddf5ffb700 1 -- 172.21.15.47:0/296423084 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdde000d600 con 0x7fddf8072520 2023-11-15T04:28:33.905 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.711+0000 7fddf5ffb700 1 -- 172.21.15.47:0/296423084 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdde000dbd0 con 0x7fddf8072520 2023-11-15T04:28:33.905 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.711+0000 7fddf5ffb700 1 -- 172.21.15.47:0/296423084 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdde0011c00 con 0x7fddf8072520 2023-11-15T04:28:33.905 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.711+0000 7fddfd79c700 1 -- 172.21.15.47:0/296423084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 msgr2=0x7fddf8070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:33.905 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.711+0000 7fddfd79c700 1 --2- 172.21.15.47:0/296423084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf8070ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fdde000b5f0 tx=0x7fdde0006ee0).stop 2023-11-15T04:28:33.906 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.712+0000 7fddfd79c700 1 -- 172.21.15.47:0/296423084 shutdown_connections 2023-11-15T04:28:33.906 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.712+0000 7fddfd79c700 1 --2- 172.21.15.47:0/296423084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf8070ac0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:33.906 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.712+0000 7fddfd79c700 1 -- 172.21.15.47:0/296423084 >> 172.21.15.47:0/296423084 conn(0x7fddf80f74b0 msgr2=0x7fddf80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:33.906 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.712+0000 7fddfd79c700 1 -- 172.21.15.47:0/296423084 shutdown_connections 2023-11-15T04:28:33.906 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.712+0000 7fddfd79c700 1 -- 172.21.15.47:0/296423084 wait complete. 2023-11-15T04:28:33.907 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.713+0000 7fddfd79c700 1 Processor -- start 2023-11-15T04:28:33.907 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.713+0000 7fddfd79c700 1 -- start start 2023-11-15T04:28:33.907 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.713+0000 7fddfd79c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf810cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:33.907 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.713+0000 7fddfd79c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf810d260 con 0x7fddf8072520 2023-11-15T04:28:33.907 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.713+0000 7fddf6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf810cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:33.908 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.713+0000 7fddf6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf810cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56906/0 (socket says 172.21.15.47:56906) 2023-11-15T04:28:33.908 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.713+0000 7fddf6ffd700 1 -- 172.21.15.47:0/148168235 learned_addr learned my addr 172.21.15.47:0/148168235 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:33.908 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.713+0000 7fddf6ffd700 1 -- 172.21.15.47:0/148168235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde000b200 con 0x7fddf8072520 2023-11-15T04:28:33.908 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.714+0000 7fddf6ffd700 1 --2- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf810cd70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fdde0002f40 tx=0x7fdde0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:33.908 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.714+0000 7fddef7fe700 1 -- 172.21.15.47:0/148168235 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdde0011690 con 0x7fddf8072520 2023-11-15T04:28:33.909 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.714+0000 7fddef7fe700 1 -- 172.21.15.47:0/148168235 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdde001b880 con 0x7fddf8072520 2023-11-15T04:28:33.909 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.714+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf810d460 con 0x7fddf8072520 2023-11-15T04:28:33.909 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.714+0000 7fddef7fe700 1 -- 172.21.15.47:0/148168235 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdde001a6b0 con 0x7fddf8072520 2023-11-15T04:28:33.909 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.714+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf810d870 con 0x7fddf8072520 2023-11-15T04:28:33.909 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.714+0000 7fddef7fe700 1 -- 172.21.15.47:0/148168235 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fdde000f070 con 0x7fddf8072520 2023-11-15T04:28:33.910 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.715+0000 7fddef7fe700 1 --2- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fddd802fdb0 0x7fddd8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:33.910 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.715+0000 7fddef7fe700 1 -- 172.21.15.47:0/148168235 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fdde0041fe0 con 0x7fddf8072520 2023-11-15T04:28:33.910 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.715+0000 7fddf67fc700 1 --2- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fddd802fdb0 0x7fddd8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:33.910 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.715+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf8045ad0 con 0x7fddf8072520 2023-11-15T04:28:33.911 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.715+0000 7fddf67fc700 1 --2- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fddd802fdb0 0x7fddd8032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fdde80096a0 tx=0x7fdde8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:33.911 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.719+0000 7fddef7fe700 1 -- 172.21.15.47:0/148168235 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdde0016070 con 0x7fddf8072520 2023-11-15T04:28:33.919 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.919+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fddf8106b70 con 0x7fddd802fdb0 2023-11-15T04:28:33.925 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.924+0000 7fddef7fe700 1 -- 172.21.15.47:0/148168235 <== mgr.14116 v2:172.21.15.47:6800/1915104696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fddf8106b70 con 0x7fddd802fdb0 2023-11-15T04:28:33.927 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.926+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fddd802fdb0 msgr2=0x7fddd8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:33.927 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.926+0000 7fddfd79c700 1 --2- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fddd802fdb0 0x7fddd8032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fdde80096a0 tx=0x7fdde8006b40).stop 2023-11-15T04:28:33.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.926+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 msgr2=0x7fddf810cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:33.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.926+0000 7fddfd79c700 1 --2- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf810cd70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fdde0002f40 tx=0x7fdde0006ee0).stop 2023-11-15T04:28:33.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.926+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 shutdown_connections 2023-11-15T04:28:33.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.926+0000 7fddfd79c700 1 --2- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fddd802fdb0 0x7fddd8032270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:33.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.926+0000 7fddfd79c700 1 --2- 172.21.15.47:0/148168235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddf8072520 0x7fddf810cd70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:33.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.927+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 >> 172.21.15.47:0/148168235 conn(0x7fddf80f74b0 msgr2=0x7fddf80f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:33.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.927+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 shutdown_connections 2023-11-15T04:28:33.930 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:33.927+0000 7fddfd79c700 1 -- 172.21.15.47:0/148168235 wait complete. 2023-11-15T04:28:35.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:33.920840+0000 mgr.smithi047.zgysuz (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:35.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:33.925134+0000 mon.smithi047 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:35.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:33.928445+0000 mon.smithi047 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:28:35.959 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-15T04:28:37.251 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.061+0000 7f38aadf7700 1 Processor -- start 2023-11-15T04:28:37.251 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.061+0000 7f38aadf7700 1 -- start start 2023-11-15T04:28:37.252 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38aadf7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a40fe950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:37.252 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38aadf7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a40fee40 con 0x7f38a4070440 2023-11-15T04:28:37.252 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38a8b93700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a40fe950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:37.253 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38a8b93700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a40fe950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49958/0 (socket says 172.21.15.47:49958) 2023-11-15T04:28:37.253 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38a8b93700 1 -- 172.21.15.47:0/3971380787 learned_addr learned my addr 172.21.15.47:0/3971380787 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:37.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38a8b93700 1 -- 172.21.15.47:0/3971380787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a40fef80 con 0x7f38a4070440 2023-11-15T04:28:37.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38a8b93700 1 --2- 172.21.15.47:0/3971380787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a40fe950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f389800b5f0 tx=0x7f3898006ee0).ready entity=mon.0 client_cookie=b6ab78b20c56c204 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:37.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38a37fe700 1 -- 172.21.15.47:0/3971380787 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f389800d600 con 0x7f38a4070440 2023-11-15T04:28:37.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38a37fe700 1 -- 172.21.15.47:0/3971380787 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f389800dbd0 con 0x7f38a4070440 2023-11-15T04:28:37.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.062+0000 7f38a37fe700 1 -- 172.21.15.47:0/3971380787 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3898011c00 con 0x7f38a4070440 2023-11-15T04:28:37.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.063+0000 7f38aadf7700 1 -- 172.21.15.47:0/3971380787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 msgr2=0x7f38a40fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:37.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.063+0000 7f38aadf7700 1 --2- 172.21.15.47:0/3971380787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a40fe950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f389800b5f0 tx=0x7f3898006ee0).stop 2023-11-15T04:28:37.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.063+0000 7f38aadf7700 1 -- 172.21.15.47:0/3971380787 shutdown_connections 2023-11-15T04:28:37.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.063+0000 7f38aadf7700 1 --2- 172.21.15.47:0/3971380787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a40fe950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:37.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.063+0000 7f38aadf7700 1 -- 172.21.15.47:0/3971380787 >> 172.21.15.47:0/3971380787 conn(0x7f38a40f63b0 msgr2=0x7f38a40f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:37.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.063+0000 7f38aadf7700 1 -- 172.21.15.47:0/3971380787 shutdown_connections 2023-11-15T04:28:37.258 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.063+0000 7f38aadf7700 1 -- 172.21.15.47:0/3971380787 wait complete. 2023-11-15T04:28:37.258 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.064+0000 7f38aadf7700 1 Processor -- start 2023-11-15T04:28:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.064+0000 7f38aadf7700 1 -- start start 2023-11-15T04:28:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.064+0000 7f38aadf7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a410aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:37.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.064+0000 7f38aadf7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a410af30 con 0x7f38a4070440 2023-11-15T04:28:37.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.064+0000 7f38a8b93700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a410aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:37.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.064+0000 7f38a8b93700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a410aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49962/0 (socket says 172.21.15.47:49962) 2023-11-15T04:28:37.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.064+0000 7f38a8b93700 1 -- 172.21.15.47:0/2309526434 learned_addr learned my addr 172.21.15.47:0/2309526434 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:37.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.064+0000 7f38a8b93700 1 -- 172.21.15.47:0/2309526434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389800b200 con 0x7f38a4070440 2023-11-15T04:28:37.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.065+0000 7f38a8b93700 1 --2- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a410aa40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f389800e040 tx=0x7f3898006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:37.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.065+0000 7f38a17fa700 1 -- 172.21.15.47:0/2309526434 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f389800d9f0 con 0x7f38a4070440 2023-11-15T04:28:37.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.065+0000 7f38a17fa700 1 -- 172.21.15.47:0/2309526434 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f389801a9f0 con 0x7f38a4070440 2023-11-15T04:28:37.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.065+0000 7f38a17fa700 1 -- 172.21.15.47:0/2309526434 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f389801b8c0 con 0x7f38a4070440 2023-11-15T04:28:37.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.065+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a410b130 con 0x7f38a4070440 2023-11-15T04:28:37.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.065+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a410b540 con 0x7f38a4070440 2023-11-15T04:28:37.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.066+0000 7f38a17fa700 1 -- 172.21.15.47:0/2309526434 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f389800f070 con 0x7f38a4070440 2023-11-15T04:28:37.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.066+0000 7f38a17fa700 1 --2- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f388802fdb0 0x7f3888032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:37.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.066+0000 7f38a17fa700 1 -- 172.21.15.47:0/2309526434 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f38980412c0 con 0x7f38a4070440 2023-11-15T04:28:37.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.066+0000 7f38a3fff700 1 --2- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f388802fdb0 0x7f3888032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:37.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.066+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a4059100 con 0x7f38a4070440 2023-11-15T04:28:37.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.066+0000 7f38a3fff700 1 --2- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f388802fdb0 0x7f3888032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f38900096a0 tx=0x7f3890006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:37.266 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.071+0000 7f38a17fa700 1 -- 172.21.15.47:0/2309526434 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3898016070 con 0x7f38a4070440 2023-11-15T04:28:37.266 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.238+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f38a41048a0 con 0x7f388802fdb0 2023-11-15T04:28:37.267 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.239+0000 7f38a17fa700 1 -- 172.21.15.47:0/2309526434 <== mgr.14116 v2:172.21.15.47:6800/1915104696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f38a41048a0 con 0x7f388802fdb0 2023-11-15T04:28:37.267 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: value unchanged 2023-11-15T04:28:37.267 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f388802fdb0 msgr2=0x7f3888032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:37.268 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 --2- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f388802fdb0 0x7f3888032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f38900096a0 tx=0x7f3890006b40).stop 2023-11-15T04:28:37.268 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 msgr2=0x7f38a410aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:37.269 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 --2- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a410aa40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f389800e040 tx=0x7f3898006ee0).stop 2023-11-15T04:28:37.269 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 shutdown_connections 2023-11-15T04:28:37.269 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 --2- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f388802fdb0 0x7f3888032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:37.270 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 --2- 172.21.15.47:0/2309526434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4070440 0x7f38a410aa40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:37.270 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 >> 172.21.15.47:0/2309526434 conn(0x7f38a40f63b0 msgr2=0x7f38a40f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:37.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 shutdown_connections 2023-11-15T04:28:37.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:37.242+0000 7f38aadf7700 1 -- 172.21.15.47:0/2309526434 wait complete. 2023-11-15T04:28:38.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:37.239992+0000 mgr.smithi047.zgysuz (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:Generating ssh key... 2023-11-15T04:28:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-15T04:28:39.088 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.087+0000 7f39f70ba700 1 Processor -- start 2023-11-15T04:28:39.089 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.087+0000 7f39f70ba700 1 -- start start 2023-11-15T04:28:39.089 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.087+0000 7f39f70ba700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:39.089 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.087+0000 7f39f70ba700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00fee10 con 0x7f39f00703f0 2023-11-15T04:28:39.089 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.087+0000 7f39f4e56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.087+0000 7f39f4e56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f00fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49966/0 (socket says 172.21.15.47:49966) 2023-11-15T04:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.087+0000 7f39f4e56700 1 -- 172.21.15.47:0/2070167377 learned_addr learned my addr 172.21.15.47:0/2070167377 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:39.092 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.088+0000 7f39f4e56700 1 -- 172.21.15.47:0/2070167377 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f00fef50 con 0x7f39f00703f0 2023-11-15T04:28:39.092 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.088+0000 7f39f4e56700 1 --2- 172.21.15.47:0/2070167377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f00fe920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f39ec00b5f0 tx=0x7f39ec006ee0).ready entity=mon.0 client_cookie=922f89519504c78d server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:39.092 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.088+0000 7f39e77fe700 1 -- 172.21.15.47:0/2070167377 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ec00d600 con 0x7f39f00703f0 2023-11-15T04:28:39.092 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.088+0000 7f39e77fe700 1 -- 172.21.15.47:0/2070167377 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f39ec00dbd0 con 0x7f39f00703f0 2023-11-15T04:28:39.093 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.088+0000 7f39e77fe700 1 -- 172.21.15.47:0/2070167377 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ec011c00 con 0x7f39f00703f0 2023-11-15T04:28:39.093 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.088+0000 7f39f70ba700 1 -- 172.21.15.47:0/2070167377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 msgr2=0x7f39f00fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:39.093 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.089+0000 7f39f70ba700 1 --2- 172.21.15.47:0/2070167377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f00fe920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f39ec00b5f0 tx=0x7f39ec006ee0).stop 2023-11-15T04:28:39.093 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.089+0000 7f39f70ba700 1 -- 172.21.15.47:0/2070167377 shutdown_connections 2023-11-15T04:28:39.093 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.089+0000 7f39f70ba700 1 --2- 172.21.15.47:0/2070167377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f00fe920 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:39.093 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.089+0000 7f39f70ba700 1 -- 172.21.15.47:0/2070167377 >> 172.21.15.47:0/2070167377 conn(0x7f39f00f63c0 msgr2=0x7f39f00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.089+0000 7f39f70ba700 1 -- 172.21.15.47:0/2070167377 shutdown_connections 2023-11-15T04:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.089+0000 7f39f70ba700 1 -- 172.21.15.47:0/2070167377 wait complete. 2023-11-15T04:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.089+0000 7f39f70ba700 1 Processor -- start 2023-11-15T04:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.089+0000 7f39f70ba700 1 -- start start 2023-11-15T04:28:39.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39f70ba700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:39.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39f70ba700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f010aeb0 con 0x7f39f00703f0 2023-11-15T04:28:39.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39f4e56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:39.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39f4e56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f010a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49972/0 (socket says 172.21.15.47:49972) 2023-11-15T04:28:39.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39f4e56700 1 -- 172.21.15.47:0/3401771456 learned_addr learned my addr 172.21.15.47:0/3401771456 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:39.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39f4e56700 1 -- 172.21.15.47:0/3401771456 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec00b200 con 0x7f39f00703f0 2023-11-15T04:28:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39f4e56700 1 --2- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f010a9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f39ec00b190 tx=0x7f39ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39e57fa700 1 -- 172.21.15.47:0/3401771456 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ec01b410 con 0x7f39f00703f0 2023-11-15T04:28:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39e57fa700 1 -- 172.21.15.47:0/3401771456 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f39ec01b9e0 con 0x7f39f00703f0 2023-11-15T04:28:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.090+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f010b0b0 con 0x7f39f00703f0 2023-11-15T04:28:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.091+0000 7f39e57fa700 1 -- 172.21.15.47:0/3401771456 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ec01a880 con 0x7f39f00703f0 2023-11-15T04:28:39.097 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.091+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f010b4c0 con 0x7f39f00703f0 2023-11-15T04:28:39.097 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.091+0000 7f39e57fa700 1 -- 172.21.15.47:0/3401771456 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f39ec00f070 con 0x7f39f00703f0 2023-11-15T04:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.092+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f0045ad0 con 0x7f39f00703f0 2023-11-15T04:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.092+0000 7f39e57fa700 1 --2- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f39dc02fd50 0x7f39dc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.092+0000 7f39e57fa700 1 -- 172.21.15.47:0/3401771456 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f39ec041370 con 0x7f39f00703f0 2023-11-15T04:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.092+0000 7f39e7fff700 1 --2- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f39dc02fd50 0x7f39dc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.092+0000 7f39e7fff700 1 --2- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f39dc02fd50 0x7f39dc032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f39e00096a0 tx=0x7f39e0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.097+0000 7f39e57fa700 1 -- 172.21.15.47:0/3401771456 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39ec010360 con 0x7f39f00703f0 2023-11-15T04:28:39.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.263+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f39f010be00 con 0x7f39dc02fd50 2023-11-15T04:28:39.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.619+0000 7f39e57fa700 1 -- 172.21.15.47:0/3401771456 <== mgr.14116 v2:172.21.15.47:6800/1915104696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f39f010be00 con 0x7f39dc02fd50 2023-11-15T04:28:39.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.620+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f39dc02fd50 msgr2=0x7f39dc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:39.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.620+0000 7f39f70ba700 1 --2- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f39dc02fd50 0x7f39dc032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f39e00096a0 tx=0x7f39e0006b40).stop 2023-11-15T04:28:39.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.621+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 msgr2=0x7f39f010a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:39.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.621+0000 7f39f70ba700 1 --2- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f010a9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f39ec00b190 tx=0x7f39ec006ee0).stop 2023-11-15T04:28:39.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.621+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 shutdown_connections 2023-11-15T04:28:39.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.621+0000 7f39f70ba700 1 --2- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f39dc02fd50 0x7f39dc032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:39.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.621+0000 7f39f70ba700 1 --2- 172.21.15.47:0/3401771456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39f00703f0 0x7f39f010a9c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:39.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.621+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 >> 172.21.15.47:0/3401771456 conn(0x7f39f00f63c0 msgr2=0x7f39f00f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:39.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.621+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 shutdown_connections 2023-11-15T04:28:39.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:39.621+0000 7f39f70ba700 1 -- 172.21.15.47:0/3401771456 wait complete. 2023-11-15T04:28:39.993 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-15T04:28:40.345 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.342+0000 7f7b88c88700 1 Processor -- start 2023-11-15T04:28:40.345 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.343+0000 7f7b88c88700 1 -- start start 2023-11-15T04:28:40.345 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.343+0000 7f7b88c88700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b84100d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.343+0000 7f7b88c88700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b840fc760 con 0x7f7b84070420 2023-11-15T04:28:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.343+0000 7f7b8259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b84100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.343+0000 7f7b8259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b84100d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49980/0 (socket says 172.21.15.47:49980) 2023-11-15T04:28:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.343+0000 7f7b8259c700 1 -- 172.21.15.47:0/4272488256 learned_addr learned my addr 172.21.15.47:0/4272488256 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.344+0000 7f7b8259c700 1 -- 172.21.15.47:0/4272488256 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b84101260 con 0x7f7b84070420 2023-11-15T04:28:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.344+0000 7f7b8259c700 1 --2- 172.21.15.47:0/4272488256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b84100d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f7b7400b5f0 tx=0x7f7b74006ee0).ready entity=mon.0 client_cookie=c581301532f60026 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.344+0000 7f7b8159a700 1 -- 172.21.15.47:0/4272488256 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b7400d600 con 0x7f7b84070420 2023-11-15T04:28:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.344+0000 7f7b8159a700 1 -- 172.21.15.47:0/4272488256 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7b7400dbd0 con 0x7f7b84070420 2023-11-15T04:28:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.344+0000 7f7b8159a700 1 -- 172.21.15.47:0/4272488256 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b74011c00 con 0x7f7b84070420 2023-11-15T04:28:40.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.344+0000 7f7b88c88700 1 -- 172.21.15.47:0/4272488256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 msgr2=0x7f7b84100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:40.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.344+0000 7f7b88c88700 1 --2- 172.21.15.47:0/4272488256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b84100d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f7b7400b5f0 tx=0x7f7b74006ee0).stop 2023-11-15T04:28:40.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.345+0000 7f7b88c88700 1 -- 172.21.15.47:0/4272488256 shutdown_connections 2023-11-15T04:28:40.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.345+0000 7f7b88c88700 1 --2- 172.21.15.47:0/4272488256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b84100d70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:40.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.345+0000 7f7b88c88700 1 -- 172.21.15.47:0/4272488256 >> 172.21.15.47:0/4272488256 conn(0x7f7b840f63b0 msgr2=0x7f7b840f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:40.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.345+0000 7f7b88c88700 1 -- 172.21.15.47:0/4272488256 shutdown_connections 2023-11-15T04:28:40.351 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.345+0000 7f7b88c88700 1 -- 172.21.15.47:0/4272488256 wait complete. 2023-11-15T04:28:40.351 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.345+0000 7f7b88c88700 1 Processor -- start 2023-11-15T04:28:40.351 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.345+0000 7f7b88c88700 1 -- start start 2023-11-15T04:28:40.352 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.345+0000 7f7b88c88700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b8410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:40.352 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.345+0000 7f7b88c88700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8410d1e0 con 0x7f7b84070420 2023-11-15T04:28:40.352 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b8259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b8410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:40.352 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b8259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b8410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49982/0 (socket says 172.21.15.47:49982) 2023-11-15T04:28:40.352 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b8259c700 1 -- 172.21.15.47:0/4209365054 learned_addr learned my addr 172.21.15.47:0/4209365054 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:40.353 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b8259c700 1 -- 172.21.15.47:0/4209365054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7400b200 con 0x7f7b84070420 2023-11-15T04:28:40.353 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b8259c700 1 --2- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b8410ccf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7b7400e040 tx=0x7f7b74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:40.353 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b7affd700 1 -- 172.21.15.47:0/4209365054 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b7400d9f0 con 0x7f7b84070420 2023-11-15T04:28:40.353 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b7affd700 1 -- 172.21.15.47:0/4209365054 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7b7401a9f0 con 0x7f7b84070420 2023-11-15T04:28:40.353 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8410d3e0 con 0x7f7b84070420 2023-11-15T04:28:40.354 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b7affd700 1 -- 172.21.15.47:0/4209365054 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b7401b8c0 con 0x7f7b84070420 2023-11-15T04:28:40.354 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.346+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8410d7f0 con 0x7f7b84070420 2023-11-15T04:28:40.354 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.347+0000 7f7b7affd700 1 -- 172.21.15.47:0/4209365054 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7b7400f070 con 0x7f7b84070420 2023-11-15T04:28:40.354 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.347+0000 7f7b7affd700 1 --2- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7b6402fdb0 0x7f7b64032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:40.354 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.347+0000 7f7b7affd700 1 -- 172.21.15.47:0/4209365054 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7b74041360 con 0x7f7b84070420 2023-11-15T04:28:40.355 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.347+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b84045ad0 con 0x7f7b84070420 2023-11-15T04:28:40.355 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.347+0000 7f7b81d9b700 1 --2- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7b6402fdb0 0x7f7b64032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:40.355 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.348+0000 7f7b81d9b700 1 --2- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7b6402fdb0 0x7f7b64032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7b6c0096a0 tx=0x7f7b6c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:40.355 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.352+0000 7f7b7affd700 1 -- 172.21.15.47:0/4209365054 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b74016070 con 0x7f7b84070420 2023-11-15T04:28:40.518 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.518+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b8410ed50 con 0x7f7b6402fdb0 2023-11-15T04:28:40.519 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.518+0000 7f7b7affd700 1 -- 172.21.15.47:0/4209365054 <== mgr.14116 v2:172.21.15.47:6800/1915104696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7b8410ed50 con 0x7f7b6402fdb0 2023-11-15T04:28:40.520 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCy0qrdUxrxnhvk+iYQwiUpKs7qnJhVAaM3q2FtgBHCoDfx8b1Os+kPjXiVCPoYC7QcpLXaiviRBNsUK7k86PYiJuHUy8Tf2oL0+A04yt0Y4SBjthSTHquqx5Ld9/bKqU6JXRHPTtSiEgyvOzkEgek37x+BawoJ3iwuGGhBigneviwea9Zs9KpexCCrCBfqsoYBdspsmIACPt7pKF5GXeaAEdtXYIRgmfqWG0F5BR++OoYoojR/gQ2eiAf2MshaIkYBvvHjeqQJ3wuOSbZQD31LqzCPGXgrZ6W4VlKNpEkFnG0sXi+RbAn4NunFamrLLZeKCGvsN2sQleJDRKbVFWwexvsf5O0kzJiO+06MAZtgkwcESJMOlRRQSuU7rI382dl5U6SzZj9o/ye/Bo27/VKSqDTu7o3GVl3p0ABeZqDG8baEBh+d8jauYGSvVzFQSbp9Q+AMsBiski+SvzvppZ8+9yV+b+v4WJ2Y88NZGQceWkGnMrY6jqe0xOXDQ/XSABE= ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:28:40.522 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7b6402fdb0 msgr2=0x7f7b64032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:40.522 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 --2- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7b6402fdb0 0x7f7b64032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7b6c0096a0 tx=0x7f7b6c006b40).stop 2023-11-15T04:28:40.522 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 msgr2=0x7f7b8410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:40.522 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 --2- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b8410ccf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7b7400e040 tx=0x7f7b74006ee0).stop 2023-11-15T04:28:40.523 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 shutdown_connections 2023-11-15T04:28:40.523 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 --2- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7b6402fdb0 0x7f7b64032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:40.523 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 --2- 172.21.15.47:0/4209365054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b84070420 0x7f7b8410ccf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:40.523 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 >> 172.21.15.47:0/4209365054 conn(0x7f7b840f63b0 msgr2=0x7f7b840f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:40.523 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 shutdown_connections 2023-11-15T04:28:40.524 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:40.520+0000 7f7b88c88700 1 -- 172.21.15.47:0/4209365054 wait complete. 2023-11-15T04:28:40.842 INFO:teuthology.orchestra.run.smithi047.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-15T04:28:40.843 INFO:teuthology.orchestra.run.smithi047.stderr:Adding key to root@localhost authorized_keys... 2023-11-15T04:28:40.844 INFO:teuthology.orchestra.run.smithi047.stderr:Adding host smithi047... 2023-11-15T04:28:40.844 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi047 172.21.15.47 2023-11-15T04:28:40.949 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:28:39.264719+0000 mgr.smithi047.zgysuz (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-11-15T04:28:40.949 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:39.617346+0000 mon.smithi047 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:40.949 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:39.619147+0000 mon.smithi047 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:41.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.204+0000 7f278a4d7700 1 Processor -- start 2023-11-15T04:28:41.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.204+0000 7f278a4d7700 1 -- start start 2023-11-15T04:28:41.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.204+0000 7f278a4d7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:41.206 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.204+0000 7f278a4d7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27840fdc40 con 0x7f27840fd330 2023-11-15T04:28:41.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.204+0000 7f2783fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:41.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.204+0000 7f2783fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49986/0 (socket says 172.21.15.47:49986) 2023-11-15T04:28:41.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.204+0000 7f2783fff700 1 -- 172.21.15.47:0/3341401876 learned_addr learned my addr 172.21.15.47:0/3341401876 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:41.207 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.204+0000 7f2783fff700 1 -- 172.21.15.47:0/3341401876 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27840fdd80 con 0x7f27840fd330 2023-11-15T04:28:41.208 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.205+0000 7f2783fff700 1 --2- 172.21.15.47:0/3341401876 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fd750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f277400b5f0 tx=0x7f2774006ee0).ready entity=mon.0 client_cookie=3a33505d94f5d286 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:41.208 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.205+0000 7f2782ffd700 1 -- 172.21.15.47:0/3341401876 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f277400d600 con 0x7f27840fd330 2023-11-15T04:28:41.211 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.205+0000 7f2782ffd700 1 -- 172.21.15.47:0/3341401876 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f277400dbd0 con 0x7f27840fd330 2023-11-15T04:28:41.211 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.205+0000 7f2782ffd700 1 -- 172.21.15.47:0/3341401876 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2774011c00 con 0x7f27840fd330 2023-11-15T04:28:41.211 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.205+0000 7f278a4d7700 1 -- 172.21.15.47:0/3341401876 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 msgr2=0x7f27840fd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:41.211 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.205+0000 7f278a4d7700 1 --2- 172.21.15.47:0/3341401876 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fd750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f277400b5f0 tx=0x7f2774006ee0).stop 2023-11-15T04:28:41.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.205+0000 7f278a4d7700 1 -- 172.21.15.47:0/3341401876 shutdown_connections 2023-11-15T04:28:41.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.205+0000 7f278a4d7700 1 --2- 172.21.15.47:0/3341401876 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fd750 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:41.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.205+0000 7f278a4d7700 1 -- 172.21.15.47:0/3341401876 >> 172.21.15.47:0/3341401876 conn(0x7f27840f5b10 msgr2=0x7f27840f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:41.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.206+0000 7f278a4d7700 1 -- 172.21.15.47:0/3341401876 shutdown_connections 2023-11-15T04:28:41.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.206+0000 7f278a4d7700 1 -- 172.21.15.47:0/3341401876 wait complete. 2023-11-15T04:28:41.212 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.206+0000 7f278a4d7700 1 Processor -- start 2023-11-15T04:28:41.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.207+0000 7f278a4d7700 1 -- start start 2023-11-15T04:28:41.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.207+0000 7f278a4d7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fcc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:41.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.207+0000 7f278a4d7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27840fb080 con 0x7f27840fd330 2023-11-15T04:28:41.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.207+0000 7f2783fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:41.213 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.207+0000 7f2783fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fcc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49996/0 (socket says 172.21.15.47:49996) 2023-11-15T04:28:41.214 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.207+0000 7f2783fff700 1 -- 172.21.15.47:0/3263303178 learned_addr learned my addr 172.21.15.47:0/3263303178 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:41.214 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.207+0000 7f2783fff700 1 -- 172.21.15.47:0/3263303178 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277400b200 con 0x7f27840fd330 2023-11-15T04:28:41.214 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.207+0000 7f2783fff700 1 --2- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fcc00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2774002f40 tx=0x7f2774006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:41.214 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.208+0000 7f2780ff9700 1 -- 172.21.15.47:0/3263303178 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f277401b410 con 0x7f27840fd330 2023-11-15T04:28:41.215 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.208+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27840fb220 con 0x7f27840fd330 2023-11-15T04:28:41.215 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.208+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27840fb630 con 0x7f27840fd330 2023-11-15T04:28:41.215 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.208+0000 7f2780ff9700 1 -- 172.21.15.47:0/3263303178 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2774006a70 con 0x7f27840fd330 2023-11-15T04:28:41.215 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.208+0000 7f2780ff9700 1 -- 172.21.15.47:0/3263303178 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f277401a6e0 con 0x7f27840fd330 2023-11-15T04:28:41.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.209+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2764004fa0 con 0x7f27840fd330 2023-11-15T04:28:41.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.209+0000 7f2780ff9700 1 -- 172.21.15.47:0/3263303178 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f277400f070 con 0x7f27840fd330 2023-11-15T04:28:41.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.209+0000 7f2780ff9700 1 --2- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f277002fd50 0x7f2770032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:41.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.209+0000 7f2780ff9700 1 -- 172.21.15.47:0/3263303178 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f27740411c0 con 0x7f27840fd330 2023-11-15T04:28:41.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.209+0000 7f27837fe700 1 --2- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f277002fd50 0x7f2770032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:41.216 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.210+0000 7f27837fe700 1 --2- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f277002fd50 0x7f2770032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f27780096a0 tx=0x7f2778006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:41.217 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.213+0000 7f2780ff9700 1 -- 172.21.15.47:0/3263303178 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2774022870 con 0x7f27840fd330 2023-11-15T04:28:41.403 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:41.402+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi047", "addr": "172.21.15.47", "target": ["mon-mgr", ""]}) v1 -- 0x7f2764000bc0 con 0x7f277002fd50 2023-11-15T04:28:41.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:41 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:40.519580+0000 mgr.smithi047.zgysuz (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:42.065 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Added host 'smithi047' 2023-11-15T04:28:42.066 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.064+0000 7f2780ff9700 1 -- 172.21.15.47:0/3263303178 <== mgr.14116 v2:172.21.15.47:6800/1915104696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f2764000bc0 con 0x7f277002fd50 2023-11-15T04:28:42.067 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.066+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f277002fd50 msgr2=0x7f2770032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:42.067 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.066+0000 7f278a4d7700 1 --2- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f277002fd50 0x7f2770032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f27780096a0 tx=0x7f2778006b40).stop 2023-11-15T04:28:42.068 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.067+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 msgr2=0x7f27840fcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:42.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.067+0000 7f278a4d7700 1 --2- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fcc00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2774002f40 tx=0x7f2774006ee0).stop 2023-11-15T04:28:42.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.067+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 shutdown_connections 2023-11-15T04:28:42.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.067+0000 7f278a4d7700 1 --2- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f277002fd50 0x7f2770032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:42.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.067+0000 7f278a4d7700 1 --2- 172.21.15.47:0/3263303178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27840fd330 0x7f27840fcc00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:42.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.067+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 >> 172.21.15.47:0/3263303178 conn(0x7f27840f5b10 msgr2=0x7f27840ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:42.070 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.067+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 shutdown_connections 2023-11-15T04:28:42.070 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.067+0000 7f278a4d7700 1 -- 172.21.15.47:0/3263303178 wait complete. 2023-11-15T04:28:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying mon service with default placement... 2023-11-15T04:28:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-15T04:28:42.749 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.747+0000 7efe192ee700 1 Processor -- start 2023-11-15T04:28:42.749 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.747+0000 7efe192ee700 1 -- start start 2023-11-15T04:28:42.749 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.748+0000 7efe192ee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe140fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:42.750 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.748+0000 7efe192ee700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe140fb4a0 con 0x7efe140fab90 2023-11-15T04:28:42.752 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.748+0000 7efe12ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe140fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:42.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.748+0000 7efe12ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe140fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50012/0 (socket says 172.21.15.47:50012) 2023-11-15T04:28:42.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.748+0000 7efe12ffd700 1 -- 172.21.15.47:0/2446227102 learned_addr learned my addr 172.21.15.47:0/2446227102 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:42.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.748+0000 7efe12ffd700 1 -- 172.21.15.47:0/2446227102 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe140fb5e0 con 0x7efe140fab90 2023-11-15T04:28:42.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.748+0000 7efe12ffd700 1 --2- 172.21.15.47:0/2446227102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe140fafb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efdfc00b5f0 tx=0x7efdfc006ee0).ready entity=mon.0 client_cookie=f3b6895f5b13603b server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:42.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe11ffb700 1 -- 172.21.15.47:0/2446227102 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdfc00d600 con 0x7efe140fab90 2023-11-15T04:28:42.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe11ffb700 1 -- 172.21.15.47:0/2446227102 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efdfc00dbd0 con 0x7efe140fab90 2023-11-15T04:28:42.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe11ffb700 1 -- 172.21.15.47:0/2446227102 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdfc011c00 con 0x7efe140fab90 2023-11-15T04:28:42.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe192ee700 1 -- 172.21.15.47:0/2446227102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 msgr2=0x7efe140fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:42.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe192ee700 1 --2- 172.21.15.47:0/2446227102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe140fafb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efdfc00b5f0 tx=0x7efdfc006ee0).stop 2023-11-15T04:28:42.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe192ee700 1 -- 172.21.15.47:0/2446227102 shutdown_connections 2023-11-15T04:28:42.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe192ee700 1 --2- 172.21.15.47:0/2446227102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe140fafb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:42.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe192ee700 1 -- 172.21.15.47:0/2446227102 >> 172.21.15.47:0/2446227102 conn(0x7efe140f63b0 msgr2=0x7efe140f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:42.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe192ee700 1 -- 172.21.15.47:0/2446227102 shutdown_connections 2023-11-15T04:28:42.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.749+0000 7efe192ee700 1 -- 172.21.15.47:0/2446227102 wait complete. 2023-11-15T04:28:42.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.750+0000 7efe192ee700 1 Processor -- start 2023-11-15T04:28:42.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.750+0000 7efe192ee700 1 -- start start 2023-11-15T04:28:42.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.750+0000 7efe192ee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe1410a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:42.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.750+0000 7efe192ee700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1410aee0 con 0x7efe140fab90 2023-11-15T04:28:42.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe12ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe1410a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe12ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe1410a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50020/0 (socket says 172.21.15.47:50020) 2023-11-15T04:28:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe12ffd700 1 -- 172.21.15.47:0/123956558 learned_addr learned my addr 172.21.15.47:0/123956558 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe12ffd700 1 -- 172.21.15.47:0/123956558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc00b200 con 0x7efe140fab90 2023-11-15T04:28:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe12ffd700 1 --2- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe1410a9f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efdfc00b1d0 tx=0x7efdfc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:42.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe0b7fe700 1 -- 172.21.15.47:0/123956558 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdfc01b410 con 0x7efe140fab90 2023-11-15T04:28:42.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe0b7fe700 1 -- 172.21.15.47:0/123956558 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efdfc01b9e0 con 0x7efe140fab90 2023-11-15T04:28:42.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1410b0e0 con 0x7efe140fab90 2023-11-15T04:28:42.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe0b7fe700 1 -- 172.21.15.47:0/123956558 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdfc01a840 con 0x7efe140fab90 2023-11-15T04:28:42.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.751+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1410b4f0 con 0x7efe140fab90 2023-11-15T04:28:42.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.752+0000 7efe0b7fe700 1 -- 172.21.15.47:0/123956558 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7efdfc00f070 con 0x7efe140fab90 2023-11-15T04:28:42.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.752+0000 7efe0b7fe700 1 --2- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7efdf402fd50 0x7efdf4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:42.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.752+0000 7efe0b7fe700 1 -- 172.21.15.47:0/123956558 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7efdfc041290 con 0x7efe140fab90 2023-11-15T04:28:42.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.752+0000 7efe127fc700 1 --2- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7efdf402fd50 0x7efdf4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:42.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.753+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe14059100 con 0x7efe140fab90 2023-11-15T04:28:42.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.753+0000 7efe127fc700 1 --2- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7efdf402fd50 0x7efdf4032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7efe040096a0 tx=0x7efe04006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:42.761 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.757+0000 7efe0b7fe700 1 -- 172.21.15.47:0/123956558 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdfc016070 con 0x7efe140fab90 2023-11-15T04:28:42.953 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.952+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7efe14104a40 con 0x7efdf402fd50 2023-11-15T04:28:42.957 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.957+0000 7efe0b7fe700 1 -- 172.21.15.47:0/123956558 <== mgr.14116 v2:172.21.15.47:6800/1915104696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7efe14104a40 con 0x7efdf402fd50 2023-11-15T04:28:42.957 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-15T04:28:42.960 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.958+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7efdf402fd50 msgr2=0x7efdf4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:42.960 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.958+0000 7efe192ee700 1 --2- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7efdf402fd50 0x7efdf4032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7efe040096a0 tx=0x7efe04006b40).stop 2023-11-15T04:28:42.960 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.958+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 msgr2=0x7efe1410a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:42.960 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.958+0000 7efe192ee700 1 --2- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe1410a9f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efdfc00b1d0 tx=0x7efdfc006ee0).stop 2023-11-15T04:28:42.960 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.959+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 shutdown_connections 2023-11-15T04:28:42.961 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.959+0000 7efe192ee700 1 --2- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7efdf402fd50 0x7efdf4032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:42.961 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.959+0000 7efe192ee700 1 --2- 172.21.15.47:0/123956558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe140fab90 0x7efe1410a9f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:42.961 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.959+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 >> 172.21.15.47:0/123956558 conn(0x7efe140f63b0 msgr2=0x7efe140f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:42.961 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.959+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 shutdown_connections 2023-11-15T04:28:42.962 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:42.959+0000 7efe192ee700 1 -- 172.21.15.47:0/123956558 wait complete. 2023-11-15T04:28:42.990 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:41.404608+0000 mgr.smithi047.zgysuz (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi047", "addr": "172.21.15.47", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:42.990 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:42.064279+0000 mon.smithi047 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:42.990 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:42.064875+0000 mon.smithi047 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:28:43.300 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying mgr service with default placement... 2023-11-15T04:28:43.301 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-15T04:28:43.666 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.664+0000 7f7bce29c700 1 Processor -- start 2023-11-15T04:28:43.666 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.664+0000 7f7bce29c700 1 -- start start 2023-11-15T04:28:43.666 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.665+0000 7f7bce29c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:43.667 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.665+0000 7f7bce29c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc80fb490 con 0x7f7bc80fab80 2023-11-15T04:28:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.665+0000 7f7bc7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.665+0000 7f7bc7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50022/0 (socket says 172.21.15.47:50022) 2023-11-15T04:28:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.665+0000 7f7bc7fff700 1 -- 172.21.15.47:0/378419444 learned_addr learned my addr 172.21.15.47:0/378419444 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.665+0000 7f7bc7fff700 1 -- 172.21.15.47:0/378419444 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc80fb5d0 con 0x7f7bc80fab80 2023-11-15T04:28:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.665+0000 7f7bc7fff700 1 --2- 172.21.15.47:0/378419444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc80fafa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f7bb800b5f0 tx=0x7f7bb8006ee0).ready entity=mon.0 client_cookie=27bfe10c2da0dd8a server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:43.669 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bc6ffd700 1 -- 172.21.15.47:0/378419444 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb800d600 con 0x7f7bc80fab80 2023-11-15T04:28:43.669 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bc6ffd700 1 -- 172.21.15.47:0/378419444 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7bb800dbd0 con 0x7f7bc80fab80 2023-11-15T04:28:43.669 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bc6ffd700 1 -- 172.21.15.47:0/378419444 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb8011c00 con 0x7f7bc80fab80 2023-11-15T04:28:43.669 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bce29c700 1 -- 172.21.15.47:0/378419444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 msgr2=0x7f7bc80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:43.669 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bce29c700 1 --2- 172.21.15.47:0/378419444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc80fafa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f7bb800b5f0 tx=0x7f7bb8006ee0).stop 2023-11-15T04:28:43.670 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bce29c700 1 -- 172.21.15.47:0/378419444 shutdown_connections 2023-11-15T04:28:43.670 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bce29c700 1 --2- 172.21.15.47:0/378419444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc80fafa0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:43.670 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bce29c700 1 -- 172.21.15.47:0/378419444 >> 172.21.15.47:0/378419444 conn(0x7f7bc80f63c0 msgr2=0x7f7bc80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:43.670 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bce29c700 1 -- 172.21.15.47:0/378419444 shutdown_connections 2023-11-15T04:28:43.671 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.666+0000 7f7bce29c700 1 -- 172.21.15.47:0/378419444 wait complete. 2023-11-15T04:28:43.671 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.667+0000 7f7bce29c700 1 Processor -- start 2023-11-15T04:28:43.673 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.667+0000 7f7bce29c700 1 -- start start 2023-11-15T04:28:43.673 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bce29c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc8109900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:43.673 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bce29c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc8109df0 con 0x7f7bc80fab80 2023-11-15T04:28:43.673 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bc7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc8109900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:43.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bc7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc8109900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50036/0 (socket says 172.21.15.47:50036) 2023-11-15T04:28:43.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bc7fff700 1 -- 172.21.15.47:0/2856176506 learned_addr learned my addr 172.21.15.47:0/2856176506 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:43.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bc7fff700 1 -- 172.21.15.47:0/2856176506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb800b200 con 0x7f7bc80fab80 2023-11-15T04:28:43.675 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bc7fff700 1 --2- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc8109900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f7bb800b1d0 tx=0x7f7bb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:43.675 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bc4ff9700 1 -- 172.21.15.47:0/2856176506 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb801b410 con 0x7f7bc80fab80 2023-11-15T04:28:43.675 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bc4ff9700 1 -- 172.21.15.47:0/2856176506 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7bb801b9e0 con 0x7f7bc80fab80 2023-11-15T04:28:43.675 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc8109ff0 con 0x7f7bc80fab80 2023-11-15T04:28:43.675 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.668+0000 7f7bc4ff9700 1 -- 172.21.15.47:0/2856176506 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb801a840 con 0x7f7bc80fab80 2023-11-15T04:28:43.676 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.669+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc810a400 con 0x7f7bc80fab80 2023-11-15T04:28:43.676 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.669+0000 7f7bc4ff9700 1 -- 172.21.15.47:0/2856176506 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7bb800f070 con 0x7f7bc80fab80 2023-11-15T04:28:43.676 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.669+0000 7f7bc4ff9700 1 --2- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7bb402fd50 0x7f7bb4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:43.676 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.669+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc8045ad0 con 0x7f7bc80fab80 2023-11-15T04:28:43.676 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.670+0000 7f7bc4ff9700 1 -- 172.21.15.47:0/2856176506 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7bb8041290 con 0x7f7bc80fab80 2023-11-15T04:28:43.676 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.670+0000 7f7bc77fe700 1 --2- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7bb402fd50 0x7f7bb4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:43.677 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.670+0000 7f7bc77fe700 1 --2- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7bb402fd50 0x7f7bb4032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7bbc0096a0 tx=0x7f7bbc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:43.677 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.674+0000 7f7bc4ff9700 1 -- 172.21.15.47:0/2856176506 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bb8016070 con 0x7f7bc80fab80 2023-11-15T04:28:43.873 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.872+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bc810ace0 con 0x7f7bb402fd50 2023-11-15T04:28:43.876 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.876+0000 7f7bc4ff9700 1 -- 172.21.15.47:0/2856176506 <== mgr.14116 v2:172.21.15.47:6800/1915104696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7bc810ace0 con 0x7f7bb402fd50 2023-11-15T04:28:43.877 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-15T04:28:43.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7bb402fd50 msgr2=0x7f7bb4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:43.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 --2- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7bb402fd50 0x7f7bb4032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7bbc0096a0 tx=0x7f7bbc006b40).stop 2023-11-15T04:28:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 msgr2=0x7f7bc8109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 --2- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc8109900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f7bb800b1d0 tx=0x7f7bb8006ee0).stop 2023-11-15T04:28:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 shutdown_connections 2023-11-15T04:28:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 --2- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f7bb402fd50 0x7f7bb4032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 --2- 172.21.15.47:0/2856176506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7bc80fab80 0x7f7bc8109900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 >> 172.21.15.47:0/2856176506 conn(0x7f7bc80f63c0 msgr2=0x7f7bc80f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 shutdown_connections 2023-11-15T04:28:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:43.878+0000 7f7bce29c700 1 -- 172.21.15.47:0/2856176506 wait complete. 2023-11-15T04:28:44.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:43 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:28:42.064607+0000 mgr.smithi047.zgysuz (mgr.14116) 9 : cephadm [INF] Added host smithi047 2023-11-15T04:28:44.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:43 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:42.957763+0000 mon.smithi047 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:44.295 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying crash service with default placement... 2023-11-15T04:28:44.295 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-15T04:28:44.676 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.675+0000 7f655cf00700 1 Processor -- start 2023-11-15T04:28:44.676 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.675+0000 7f655cf00700 1 -- start start 2023-11-15T04:28:44.677 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.675+0000 7f655cf00700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f65580fe950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:44.677 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.675+0000 7f655cf00700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65580fee40 con 0x7f6558070440 2023-11-15T04:28:44.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.675+0000 7f655659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f65580fe950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:44.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.675+0000 7f655659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f65580fe950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43898/0 (socket says 172.21.15.47:43898) 2023-11-15T04:28:44.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.675+0000 7f655659c700 1 -- 172.21.15.47:0/3071159948 learned_addr learned my addr 172.21.15.47:0/3071159948 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:44.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.676+0000 7f655659c700 1 -- 172.21.15.47:0/3071159948 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65580fef80 con 0x7f6558070440 2023-11-15T04:28:44.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.676+0000 7f655659c700 1 --2- 172.21.15.47:0/3071159948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f65580fe950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f654800b5f0 tx=0x7f6548006ee0).ready entity=mon.0 client_cookie=a2ed824a22563bb4 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:44.680 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.678+0000 7f655559a700 1 -- 172.21.15.47:0/3071159948 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f654800d600 con 0x7f6558070440 2023-11-15T04:28:44.680 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.678+0000 7f655559a700 1 -- 172.21.15.47:0/3071159948 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f654800dbd0 con 0x7f6558070440 2023-11-15T04:28:44.680 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.678+0000 7f655559a700 1 -- 172.21.15.47:0/3071159948 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6548011c00 con 0x7f6558070440 2023-11-15T04:28:44.680 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.679+0000 7f655cf00700 1 -- 172.21.15.47:0/3071159948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 msgr2=0x7f65580fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:44.681 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.679+0000 7f655cf00700 1 --2- 172.21.15.47:0/3071159948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f65580fe950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f654800b5f0 tx=0x7f6548006ee0).stop 2023-11-15T04:28:44.681 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.679+0000 7f655cf00700 1 -- 172.21.15.47:0/3071159948 shutdown_connections 2023-11-15T04:28:44.681 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.679+0000 7f655cf00700 1 --2- 172.21.15.47:0/3071159948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f65580fe950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:44.684 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.679+0000 7f655cf00700 1 -- 172.21.15.47:0/3071159948 >> 172.21.15.47:0/3071159948 conn(0x7f65580f63b0 msgr2=0x7f65580f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:44.684 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.679+0000 7f655cf00700 1 -- 172.21.15.47:0/3071159948 shutdown_connections 2023-11-15T04:28:44.684 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.679+0000 7f655cf00700 1 -- 172.21.15.47:0/3071159948 wait complete. 2023-11-15T04:28:44.684 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.680+0000 7f655cf00700 1 Processor -- start 2023-11-15T04:28:44.684 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.680+0000 7f655cf00700 1 -- start start 2023-11-15T04:28:44.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.680+0000 7f655cf00700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f655810a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:44.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.680+0000 7f655cf00700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655810aeb0 con 0x7f6558070440 2023-11-15T04:28:44.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.680+0000 7f655659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f655810a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:44.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.680+0000 7f655659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f655810a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43906/0 (socket says 172.21.15.47:43906) 2023-11-15T04:28:44.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.680+0000 7f655659c700 1 -- 172.21.15.47:0/1046661110 learned_addr learned my addr 172.21.15.47:0/1046661110 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:44.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.681+0000 7f655659c700 1 -- 172.21.15.47:0/1046661110 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654800b200 con 0x7f6558070440 2023-11-15T04:28:44.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.681+0000 7f655659c700 1 --2- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f655810a9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f654800b190 tx=0x7f6548006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:44.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.681+0000 7f6546ffd700 1 -- 172.21.15.47:0/1046661110 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f654801b410 con 0x7f6558070440 2023-11-15T04:28:44.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.681+0000 7f6546ffd700 1 -- 172.21.15.47:0/1046661110 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f654801b9e0 con 0x7f6558070440 2023-11-15T04:28:44.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.681+0000 7f6546ffd700 1 -- 172.21.15.47:0/1046661110 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f654801a880 con 0x7f6558070440 2023-11-15T04:28:44.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.681+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655810b0b0 con 0x7f6558070440 2023-11-15T04:28:44.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.681+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655810b4c0 con 0x7f6558070440 2023-11-15T04:28:44.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.682+0000 7f6546ffd700 1 -- 172.21.15.47:0/1046661110 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f654800f070 con 0x7f6558070440 2023-11-15T04:28:44.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.682+0000 7f6546ffd700 1 --2- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f653c02fdb0 0x7f653c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:44.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.682+0000 7f6546ffd700 1 -- 172.21.15.47:0/1046661110 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f65480411e0 con 0x7f6558070440 2023-11-15T04:28:44.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.682+0000 7f6555d9b700 1 --2- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f653c02fdb0 0x7f653c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:44.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.682+0000 7f6555d9b700 1 --2- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f653c02fdb0 0x7f653c032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f654c0096a0 tx=0x7f654c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:44.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.683+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6558059100 con 0x7f6558070440 2023-11-15T04:28:44.689 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.687+0000 7f6546ffd700 1 -- 172.21.15.47:0/1046661110 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6548016070 con 0x7f6558070440 2023-11-15T04:28:44.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.879+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f65581048a0 con 0x7f653c02fdb0 2023-11-15T04:28:44.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.883+0000 7f6546ffd700 1 -- 172.21.15.47:0/1046661110 <== mgr.14116 v2:172.21.15.47:6800/1915104696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f65581048a0 con 0x7f653c02fdb0 2023-11-15T04:28:44.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-15T04:28:44.886 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f653c02fdb0 msgr2=0x7f653c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:44.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 --2- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f653c02fdb0 0x7f653c032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f654c0096a0 tx=0x7f654c006b40).stop 2023-11-15T04:28:44.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 msgr2=0x7f655810a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:44.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 --2- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f655810a9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f654800b190 tx=0x7f6548006ee0).stop 2023-11-15T04:28:44.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 shutdown_connections 2023-11-15T04:28:44.888 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 --2- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f653c02fdb0 0x7f653c032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:44.888 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 --2- 172.21.15.47:0/1046661110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6558070440 0x7f655810a9c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:44.888 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 >> 172.21.15.47:0/1046661110 conn(0x7f65580f63b0 msgr2=0x7f65580f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:44.888 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 shutdown_connections 2023-11-15T04:28:44.888 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:44.885+0000 7f655cf00700 1 -- 172.21.15.47:0/1046661110 wait complete. 2023-11-15T04:28:45.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:42.954545+0000 mgr.smithi047.zgysuz (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:45.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:28:42.955396+0000 mgr.smithi047.zgysuz (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-15T04:28:45.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:43.876922+0000 mon.smithi047 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:45.217 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling mgr prometheus module... 2023-11-15T04:28:45.217 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-15T04:28:45.571 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.569+0000 7feaf86b2700 1 Processor -- start 2023-11-15T04:28:45.571 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.569+0000 7feaf86b2700 1 -- start start 2023-11-15T04:28:45.571 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.570+0000 7feaf86b2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:45.572 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.570+0000 7feaf86b2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf00fb490 con 0x7feaf00fab80 2023-11-15T04:28:45.572 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.570+0000 7feaf644e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:45.572 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.570+0000 7feaf644e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43920/0 (socket says 172.21.15.47:43920) 2023-11-15T04:28:45.572 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.570+0000 7feaf644e700 1 -- 172.21.15.47:0/4184180333 learned_addr learned my addr 172.21.15.47:0/4184180333 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:45.575 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.570+0000 7feaf644e700 1 -- 172.21.15.47:0/4184180333 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf00fb5d0 con 0x7feaf00fab80 2023-11-15T04:28:45.576 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.570+0000 7feaf644e700 1 --2- 172.21.15.47:0/4184180333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf00fafa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7feaec00b870 tx=0x7feaec006f70).ready entity=mon.0 client_cookie=ff6cd53586941bc0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:45.576 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.570+0000 7feaf544c700 1 -- 172.21.15.47:0/4184180333 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaec00da20 con 0x7feaf00fab80 2023-11-15T04:28:45.576 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.571+0000 7feaf544c700 1 -- 172.21.15.47:0/4184180333 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feaec010470 con 0x7feaf00fab80 2023-11-15T04:28:45.576 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.571+0000 7feaf544c700 1 -- 172.21.15.47:0/4184180333 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaec00f5b0 con 0x7feaf00fab80 2023-11-15T04:28:45.576 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.571+0000 7feaf86b2700 1 -- 172.21.15.47:0/4184180333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 msgr2=0x7feaf00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:45.577 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.571+0000 7feaf86b2700 1 --2- 172.21.15.47:0/4184180333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf00fafa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7feaec00b870 tx=0x7feaec006f70).stop 2023-11-15T04:28:45.577 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.571+0000 7feaf86b2700 1 -- 172.21.15.47:0/4184180333 shutdown_connections 2023-11-15T04:28:45.577 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.571+0000 7feaf86b2700 1 --2- 172.21.15.47:0/4184180333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf00fafa0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:45.577 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.571+0000 7feaf86b2700 1 -- 172.21.15.47:0/4184180333 >> 172.21.15.47:0/4184180333 conn(0x7feaf00f63c0 msgr2=0x7feaf00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:45.577 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.571+0000 7feaf86b2700 1 -- 172.21.15.47:0/4184180333 shutdown_connections 2023-11-15T04:28:45.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.571+0000 7feaf86b2700 1 -- 172.21.15.47:0/4184180333 wait complete. 2023-11-15T04:28:45.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.572+0000 7feaf86b2700 1 Processor -- start 2023-11-15T04:28:45.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.572+0000 7feaf86b2700 1 -- start start 2023-11-15T04:28:45.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.572+0000 7feaf86b2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf0108780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:45.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.572+0000 7feaf86b2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0108c70 con 0x7feaf00fab80 2023-11-15T04:28:45.578 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.572+0000 7feaf644e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf0108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:45.579 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.572+0000 7feaf644e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf0108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43926/0 (socket says 172.21.15.47:43926) 2023-11-15T04:28:45.579 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.572+0000 7feaf644e700 1 -- 172.21.15.47:0/2849555924 learned_addr learned my addr 172.21.15.47:0/2849555924 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:45.579 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.573+0000 7feaf644e700 1 -- 172.21.15.47:0/2849555924 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec00b480 con 0x7feaf00fab80 2023-11-15T04:28:45.579 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.573+0000 7feaf644e700 1 --2- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf0108780 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7feaec006800 tx=0x7feaec006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:45.579 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.573+0000 7feadeffd700 1 -- 172.21.15.47:0/2849555924 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaec00f820 con 0x7feaf00fab80 2023-11-15T04:28:45.580 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.573+0000 7feadeffd700 1 -- 172.21.15.47:0/2849555924 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feaec00fdf0 con 0x7feaf00fab80 2023-11-15T04:28:45.580 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.573+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf0108e70 con 0x7feaf00fab80 2023-11-15T04:28:45.580 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.573+0000 7feadeffd700 1 -- 172.21.15.47:0/2849555924 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaec019d50 con 0x7feaf00fab80 2023-11-15T04:28:45.580 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.573+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf0109280 con 0x7feaf00fab80 2023-11-15T04:28:45.580 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.574+0000 7feadeffd700 1 -- 172.21.15.47:0/2849555924 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7feaec016030 con 0x7feaf00fab80 2023-11-15T04:28:45.581 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.574+0000 7feadeffd700 1 --2- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fead802fdb0 0x7fead8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:45.581 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.574+0000 7feadeffd700 1 -- 172.21.15.47:0/2849555924 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7feaec043f40 con 0x7feaf00fab80 2023-11-15T04:28:45.581 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.574+0000 7feaf5c4d700 1 --2- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fead802fdb0 0x7fead8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:45.581 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.574+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf0059100 con 0x7feaf00fab80 2023-11-15T04:28:45.582 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.575+0000 7feaf5c4d700 1 --2- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fead802fdb0 0x7fead8032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7feae00096a0 tx=0x7feae0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:45.582 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.579+0000 7feadeffd700 1 -- 172.21.15.47:0/2849555924 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feaec014080 con 0x7feaf00fab80 2023-11-15T04:28:45.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:45.784+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7feaf0045ad0 con 0x7feaf00fab80 2023-11-15T04:28:46.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:45 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:43.873974+0000 mgr.smithi047.zgysuz (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:46.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:45 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:28:43.874531+0000 mgr.smithi047.zgysuz (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-15T04:28:46.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:45 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:44.884233+0000 mon.smithi047 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.47:0/1193069690' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:46.680 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.679+0000 7feadeffd700 1 -- 172.21.15.47:0/2849555924 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7feaec019550 con 0x7feaf00fab80 2023-11-15T04:28:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.681+0000 7feadeffd700 1 -- 172.21.15.47:0/2849555924 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7feaec005d50 con 0x7feaf00fab80 2023-11-15T04:28:46.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fead802fdb0 msgr2=0x7fead8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:46.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 --2- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fead802fdb0 0x7fead8032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7feae00096a0 tx=0x7feae0006b40).stop 2023-11-15T04:28:46.685 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 msgr2=0x7feaf0108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:46.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 --2- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf0108780 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7feaec006800 tx=0x7feaec006f70).stop 2023-11-15T04:28:46.686 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 shutdown_connections 2023-11-15T04:28:46.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 --2- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7fead802fdb0 0x7fead8032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:46.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 --2- 172.21.15.47:0/2849555924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00fab80 0x7feaf0108780 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:46.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 >> 172.21.15.47:0/2849555924 conn(0x7feaf00f63c0 msgr2=0x7feaf00ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:46.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 shutdown_connections 2023-11-15T04:28:46.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:46.684+0000 7feaf86b2700 1 -- 172.21.15.47:0/2849555924 wait complete. 2023-11-15T04:28:46.994 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:44.881213+0000 mgr.smithi047.zgysuz (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:46.994 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:28:44.881929+0000 mgr.smithi047.zgysuz (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-15T04:28:46.994 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:45.785737+0000 mon.smithi047 (mon.0) 60 : audit [INF] from='client.? 172.21.15.47:0/2849555924' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-15T04:28:47.049 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying prometheus service with default placement... 2023-11-15T04:28:47.050 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-15T04:28:47.417 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.416+0000 7f8578c49700 1 Processor -- start 2023-11-15T04:28:47.417 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.416+0000 7f8578c49700 1 -- start start 2023-11-15T04:28:47.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.416+0000 7f8578c49700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f85740fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:47.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.416+0000 7f8578c49700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85740fedd0 con 0x7f8574070430 2023-11-15T04:28:47.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.416+0000 7f857259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f85740fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:47.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.416+0000 7f857259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f85740fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43948/0 (socket says 172.21.15.47:43948) 2023-11-15T04:28:47.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.416+0000 7f857259c700 1 -- 172.21.15.47:0/2480182405 learned_addr learned my addr 172.21.15.47:0/2480182405 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:47.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.417+0000 7f857259c700 1 -- 172.21.15.47:0/2480182405 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85740fef10 con 0x7f8574070430 2023-11-15T04:28:47.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.417+0000 7f857259c700 1 --2- 172.21.15.47:0/2480182405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f85740fe8e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f856400b5f0 tx=0x7f8564006ee0).ready entity=mon.0 client_cookie=e10af74d29bb9c3b server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:47.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.417+0000 7f857159a700 1 -- 172.21.15.47:0/2480182405 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f856400d600 con 0x7f8574070430 2023-11-15T04:28:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.417+0000 7f857159a700 1 -- 172.21.15.47:0/2480182405 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f856400dbd0 con 0x7f8574070430 2023-11-15T04:28:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.417+0000 7f857159a700 1 -- 172.21.15.47:0/2480182405 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8564011c00 con 0x7f8574070430 2023-11-15T04:28:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.418+0000 7f8578c49700 1 -- 172.21.15.47:0/2480182405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 msgr2=0x7f85740fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.418+0000 7f8578c49700 1 --2- 172.21.15.47:0/2480182405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f85740fe8e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f856400b5f0 tx=0x7f8564006ee0).stop 2023-11-15T04:28:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.418+0000 7f8578c49700 1 -- 172.21.15.47:0/2480182405 shutdown_connections 2023-11-15T04:28:47.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.418+0000 7f8578c49700 1 --2- 172.21.15.47:0/2480182405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f85740fe8e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:47.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.418+0000 7f8578c49700 1 -- 172.21.15.47:0/2480182405 >> 172.21.15.47:0/2480182405 conn(0x7f85740f63c0 msgr2=0x7f85740f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:47.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.418+0000 7f8578c49700 1 -- 172.21.15.47:0/2480182405 shutdown_connections 2023-11-15T04:28:47.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.418+0000 7f8578c49700 1 -- 172.21.15.47:0/2480182405 wait complete. 2023-11-15T04:28:47.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.419+0000 7f8578c49700 1 Processor -- start 2023-11-15T04:28:47.423 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.419+0000 7f8578c49700 1 -- start start 2023-11-15T04:28:47.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.419+0000 7f8578c49700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f857410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:47.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.419+0000 7f8578c49700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857410aef0 con 0x7f8574070430 2023-11-15T04:28:47.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.419+0000 7f857259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f857410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:47.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.419+0000 7f857259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f857410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43954/0 (socket says 172.21.15.47:43954) 2023-11-15T04:28:47.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.419+0000 7f857259c700 1 -- 172.21.15.47:0/3648242846 learned_addr learned my addr 172.21.15.47:0/3648242846 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:47.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.420+0000 7f857259c700 1 -- 172.21.15.47:0/3648242846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856400b200 con 0x7f8574070430 2023-11-15T04:28:47.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.420+0000 7f857259c700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f857410aa00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f856400b190 tx=0x7f8564006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:47.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.420+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f856401b410 con 0x7f8574070430 2023-11-15T04:28:47.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.420+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f856401b9e0 con 0x7f8574070430 2023-11-15T04:28:47.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.420+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857410b0f0 con 0x7f8574070430 2023-11-15T04:28:47.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.420+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f856401a880 con 0x7f8574070430 2023-11-15T04:28:47.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.420+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857410b500 con 0x7f8574070430 2023-11-15T04:28:47.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.421+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f856400f070 con 0x7f8574070430 2023-11-15T04:28:47.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.421+0000 7f8562ffd700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 0x7f85580322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:47.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.421+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8564041270 con 0x7f8574070430 2023-11-15T04:28:47.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.421+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8574045ad0 con 0x7f8574070430 2023-11-15T04:28:47.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.421+0000 7f8571d9b700 1 -- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 msgr2=0x7f85580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1915104696 2023-11-15T04:28:47.428 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.421+0000 7f8571d9b700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 0x7f85580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:28:47.429 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.426+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8564016070 con 0x7f8574070430 2023-11-15T04:28:47.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.619+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 --> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f8574104960 con 0x7f855802fe00 2023-11-15T04:28:47.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.622+0000 7f8571d9b700 1 -- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 msgr2=0x7f85580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1915104696 2023-11-15T04:28:47.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:47.622+0000 7f8571d9b700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 0x7f85580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:28:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:46.682303+0000 mon.smithi047 (mon.0) 61 : audit [INF] from='client.? 172.21.15.47:0/2849555924' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-15T04:28:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:46.682369+0000 mon.smithi047 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi047.zgysuz(active, since 17s) 2023-11-15T04:28:48.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:48.023+0000 7f8571d9b700 1 -- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 msgr2=0x7f85580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1915104696 2023-11-15T04:28:48.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:48.023+0000 7f8571d9b700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 0x7f85580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T04:28:48.825 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:48.824+0000 7f8571d9b700 1 -- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 msgr2=0x7f85580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1915104696 2023-11-15T04:28:48.825 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:48.824+0000 7f8571d9b700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 0x7f85580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T04:28:50.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:50.425+0000 7f8571d9b700 1 -- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 msgr2=0x7f85580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1915104696 2023-11-15T04:28:50.427 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:50.425+0000 7f8571d9b700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 0x7f85580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T04:28:52.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:52.678+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f8564025460 con 0x7f8574070430 2023-11-15T04:28:52.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:52.678+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 msgr2=0x7f85580322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:28:52.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:52.678+0000 7f8562ffd700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 0x7f85580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:28:53.681 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.679+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f8564043210 con 0x7f8574070430 2023-11-15T04:28:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.680+0000 7f8562ffd700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f8558033c20 0x7f8558036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.680+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 --> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f8574104960 con 0x7f8558033c20 2023-11-15T04:28:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.680+0000 7f8571d9b700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f8558033c20 0x7f8558036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:53.683 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.681+0000 7f8571d9b700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f8558033c20 0x7f8558036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8568006fd0 tx=0x7f8568006d30).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:53.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.686+0000 7f8562ffd700 1 -- 172.21.15.47:0/3648242846 <== mgr.14144 v2:172.21.15.47:6800/826374076 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f8574104960 con 0x7f8558033c20 2023-11-15T04:28:53.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-15T04:28:53.690 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f8558033c20 msgr2=0x7f8558036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:53.690 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f8558033c20 0x7f8558036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8568006fd0 tx=0x7f8568006d30).stop 2023-11-15T04:28:53.690 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 msgr2=0x7f857410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:53.691 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f857410aa00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f856400b190 tx=0x7f8564006ee0).stop 2023-11-15T04:28:53.691 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 shutdown_connections 2023-11-15T04:28:53.691 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f8558033c20 0x7f8558036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:53.691 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8574070430 0x7f857410aa00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:53.691 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 --2- 172.21.15.47:0/3648242846 >> [v2:172.21.15.47:6800/1915104696,v1:172.21.15.47:6801/1915104696] conn(0x7f855802fe00 0x7f85580322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:28:53.692 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 >> 172.21.15.47:0/3648242846 conn(0x7f85740f63c0 msgr2=0x7f85740f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:53.692 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.688+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 shutdown_connections 2023-11-15T04:28:53.692 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:53.689+0000 7f8578c49700 1 -- 172.21.15.47:0/3648242846 wait complete. 2023-11-15T04:28:53.717 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:52.329922+0000 mon.smithi047 (mon.0) 63 : cluster [INF] Active manager daemon smithi047.zgysuz restarted 2023-11-15T04:28:53.718 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:52.330502+0000 mon.smithi047 (mon.0) 64 : cluster [INF] Activating manager daemon smithi047.zgysuz 2023-11-15T04:28:53.718 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:52.332325+0000 mon.smithi047 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-15T04:28:53.718 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:52.680298+0000 mon.smithi047 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi047.zgysuz(active, starting, since 0.349929s) 2023-11-15T04:28:53.718 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:52.681616+0000 mon.smithi047 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2023-11-15T04:28:53.718 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:52.682455+0000 mon.smithi047 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mgr metadata", "who": "smithi047.zgysuz", "id": "smithi047.zgysuz"}]: dispatch 2023-11-15T04:28:53.719 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:52.683393+0000 mon.smithi047 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:28:53.719 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:52.683578+0000 mon.smithi047 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:28:53.719 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:52.683741+0000 mon.smithi047 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:28:53.719 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:52.698023+0000 mon.smithi047 (mon.0) 72 : cluster [INF] Manager daemon smithi047.zgysuz is now available 2023-11-15T04:28:53.719 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:52.815703+0000 mon.smithi047 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:28:53.720 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:52.817365+0000 mon.smithi047 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:28:53.720 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:52.835295+0000 mon.smithi047 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:28:53.720 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:52.845108+0000 mon.smithi047 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:28:54.003 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying grafana service with default placement... 2023-11-15T04:28:54.004 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-15T04:28:54.381 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.379+0000 7f0b312db700 1 Processor -- start 2023-11-15T04:28:54.382 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.379+0000 7f0b312db700 1 -- start start 2023-11-15T04:28:54.382 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.379+0000 7f0b312db700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:54.382 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.379+0000 7f0b312db700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0fb2e0 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.382 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.379+0000 7f0b2bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:54.382 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.380+0000 7f0b2bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43994/0 (socket says 172.21.15.47:43994) 2023-11-15T04:28:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.380+0000 7f0b2bfff700 1 -- 172.21.15.47:0/1815719996 learned_addr learned my addr 172.21.15.47:0/1815719996 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.380+0000 7f0b2bfff700 1 -- 172.21.15.47:0/1815719996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0fb420 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.380+0000 7f0b2bfff700 1 --2- 172.21.15.47:0/1815719996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c0fadf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0b1c00b5f0 tx=0x7f0b1c006ee0).ready entity=mon.0 client_cookie=93c9e9420abc9637 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.380+0000 7f0b2affd700 1 -- 172.21.15.47:0/1815719996 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b1c00d600 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.380+0000 7f0b2affd700 1 -- 172.21.15.47:0/1815719996 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0b1c00dbd0 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.380+0000 7f0b2affd700 1 -- 172.21.15.47:0/1815719996 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b1c011c00 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.381+0000 7f0b312db700 1 -- 172.21.15.47:0/1815719996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 msgr2=0x7f0b2c0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.381+0000 7f0b312db700 1 --2- 172.21.15.47:0/1815719996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c0fadf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0b1c00b5f0 tx=0x7f0b1c006ee0).stop 2023-11-15T04:28:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.381+0000 7f0b312db700 1 -- 172.21.15.47:0/1815719996 shutdown_connections 2023-11-15T04:28:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.381+0000 7f0b312db700 1 --2- 172.21.15.47:0/1815719996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c0fadf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.381+0000 7f0b312db700 1 -- 172.21.15.47:0/1815719996 >> 172.21.15.47:0/1815719996 conn(0x7f0b2c0f6210 msgr2=0x7f0b2c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.381+0000 7f0b312db700 1 -- 172.21.15.47:0/1815719996 shutdown_connections 2023-11-15T04:28:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.381+0000 7f0b312db700 1 -- 172.21.15.47:0/1815719996 wait complete. 2023-11-15T04:28:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.382+0000 7f0b312db700 1 Processor -- start 2023-11-15T04:28:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.382+0000 7f0b312db700 1 -- start start 2023-11-15T04:28:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.382+0000 7f0b312db700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c10ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.382+0000 7f0b312db700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c10cf40 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b2bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c10ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b2bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c10ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44006/0 (socket says 172.21.15.47:44006) 2023-11-15T04:28:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b2bfff700 1 -- 172.21.15.47:0/2276144791 learned_addr learned my addr 172.21.15.47:0/2276144791 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:54.390 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b2bfff700 1 -- 172.21.15.47:0/2276144791 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c00b200 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.390 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b2bfff700 1 --2- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c10ca50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f0b1c00e040 tx=0x7f0b1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:54.390 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b28ff9700 1 -- 172.21.15.47:0/2276144791 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b1c01b410 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.390 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c10d140 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.390 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b28ff9700 1 -- 172.21.15.47:0/2276144791 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0b1c01b9e0 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b28ff9700 1 -- 172.21.15.47:0/2276144791 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b1c01a820 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.383+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c10d550 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.384+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c045ad0 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.384+0000 7f0b28ff9700 1 -- 172.21.15.47:0/2276144791 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f0b1c00f070 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.385+0000 7f0b28ff9700 1 --2- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f0b1802fcf0 0x7f0b180321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.385+0000 7f0b28ff9700 1 -- 172.21.15.47:0/2276144791 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0b1c041440 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.385+0000 7f0b2b7fe700 1 --2- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f0b1802fcf0 0x7f0b180321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.385+0000 7f0b2b7fe700 1 --2- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f0b1802fcf0 0x7f0b180321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0b200096a0 tx=0x7f0b20006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.389+0000 7f0b28ff9700 1 -- 172.21.15.47:0/2276144791 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b1c0224f0 con 0x7f0b2c0fa9d0 2023-11-15T04:28:54.586 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.585+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 --> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b2c10de90 con 0x7f0b1802fcf0 2023-11-15T04:28:54.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.589+0000 7f0b28ff9700 1 -- 172.21.15.47:0/2276144791 <== mgr.14144 v2:172.21.15.47:6800/826374076 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f0b2c10de90 con 0x7f0b1802fcf0 2023-11-15T04:28:54.590 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-15T04:28:54.592 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.590+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f0b1802fcf0 msgr2=0x7f0b180321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:54.592 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.590+0000 7f0b312db700 1 --2- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f0b1802fcf0 0x7f0b180321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0b200096a0 tx=0x7f0b20006b40).stop 2023-11-15T04:28:54.592 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.590+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 msgr2=0x7f0b2c10ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:54.593 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.590+0000 7f0b312db700 1 --2- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c10ca50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f0b1c00e040 tx=0x7f0b1c006ee0).stop 2023-11-15T04:28:54.593 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.591+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 shutdown_connections 2023-11-15T04:28:54.593 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.591+0000 7f0b312db700 1 --2- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f0b1802fcf0 0x7f0b180321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:54.593 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.591+0000 7f0b312db700 1 --2- 172.21.15.47:0/2276144791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b2c0fa9d0 0x7f0b2c10ca50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:54.593 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.591+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 >> 172.21.15.47:0/2276144791 conn(0x7f0b2c0f6210 msgr2=0x7f0b2c0ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:54.594 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.591+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 shutdown_connections 2023-11-15T04:28:54.594 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:54.591+0000 7f0b312db700 1 -- 172.21.15.47:0/2276144791 wait complete. 2023-11-15T04:28:54.869 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:53.683219+0000 mon.smithi047 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi047.zgysuz(active, since 1.35285s) 2023-11-15T04:28:54.869 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:28:53.684466+0000 mgr.smithi047.zgysuz (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-15T04:28:54.869 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:53.686886+0000 mon.smithi047 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:54.869 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:54.589727+0000 mon.smithi047 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:54.916 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying node-exporter service with default placement... 2023-11-15T04:28:54.916 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-15T04:28:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.299+0000 7f83404cd700 1 Processor -- start 2023-11-15T04:28:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.300+0000 7f83404cd700 1 -- start start 2023-11-15T04:28:55.302 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.300+0000 7f83404cd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:55.302 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.300+0000 7f83404cd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380fb490 con 0x7f83380fab80 2023-11-15T04:28:55.302 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.300+0000 7f833e269700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:55.302 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.300+0000 7f833e269700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57384/0 (socket says 172.21.15.47:57384) 2023-11-15T04:28:55.302 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.300+0000 7f833e269700 1 -- 172.21.15.47:0/649399709 learned_addr learned my addr 172.21.15.47:0/649399709 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:55.306 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.301+0000 7f833e269700 1 -- 172.21.15.47:0/649399709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380fb5d0 con 0x7f83380fab80 2023-11-15T04:28:55.306 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.301+0000 7f833e269700 1 --2- 172.21.15.47:0/649399709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f833400b870 tx=0x7f8334006f70).ready entity=mon.0 client_cookie=4ebc2ada8c538a05 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:55.306 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.301+0000 7f833d267700 1 -- 172.21.15.47:0/649399709 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f833400da20 con 0x7f83380fab80 2023-11-15T04:28:55.306 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.301+0000 7f833d267700 1 -- 172.21.15.47:0/649399709 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8334010470 con 0x7f83380fab80 2023-11-15T04:28:55.307 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.301+0000 7f833d267700 1 -- 172.21.15.47:0/649399709 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f833400f5b0 con 0x7f83380fab80 2023-11-15T04:28:55.307 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.301+0000 7f83404cd700 1 -- 172.21.15.47:0/649399709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 msgr2=0x7f83380fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:55.307 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.301+0000 7f83404cd700 1 --2- 172.21.15.47:0/649399709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f833400b870 tx=0x7f8334006f70).stop 2023-11-15T04:28:55.307 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.302+0000 7f83404cd700 1 -- 172.21.15.47:0/649399709 shutdown_connections 2023-11-15T04:28:55.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.302+0000 7f83404cd700 1 --2- 172.21.15.47:0/649399709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:55.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.302+0000 7f83404cd700 1 -- 172.21.15.47:0/649399709 >> 172.21.15.47:0/649399709 conn(0x7f83380f63c0 msgr2=0x7f83380f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:55.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.302+0000 7f83404cd700 1 -- 172.21.15.47:0/649399709 shutdown_connections 2023-11-15T04:28:55.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.302+0000 7f83404cd700 1 -- 172.21.15.47:0/649399709 wait complete. 2023-11-15T04:28:55.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.302+0000 7f83404cd700 1 Processor -- start 2023-11-15T04:28:55.309 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.302+0000 7f83404cd700 1 -- start start 2023-11-15T04:28:55.309 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f83404cd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f8338108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:55.309 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f83404cd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338108c60 con 0x7f83380fab80 2023-11-15T04:28:55.309 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f833e269700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f8338108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:55.312 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f833e269700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f8338108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57386/0 (socket says 172.21.15.47:57386) 2023-11-15T04:28:55.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f833e269700 1 -- 172.21.15.47:0/2331071721 learned_addr learned my addr 172.21.15.47:0/2331071721 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:55.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f833e269700 1 -- 172.21.15.47:0/2331071721 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833400b480 con 0x7f83380fab80 2023-11-15T04:28:55.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f833e269700 1 --2- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f8338108770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8334006800 tx=0x7f8334006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:55.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f8326ffd700 1 -- 172.21.15.47:0/2331071721 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f833400f820 con 0x7f83380fab80 2023-11-15T04:28:55.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f8326ffd700 1 -- 172.21.15.47:0/2331071721 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f833400fdf0 con 0x7f83380fab80 2023-11-15T04:28:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8338108e60 con 0x7f83380fab80 2023-11-15T04:28:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f8326ffd700 1 -- 172.21.15.47:0/2331071721 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8334019d50 con 0x7f83380fab80 2023-11-15T04:28:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.303+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8338109270 con 0x7f83380fab80 2023-11-15T04:28:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.304+0000 7f8326ffd700 1 -- 172.21.15.47:0/2331071721 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f8334016030 con 0x7f83380fab80 2023-11-15T04:28:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.304+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8338045ad0 con 0x7f83380fab80 2023-11-15T04:28:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.304+0000 7f8326ffd700 1 --2- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f832002fe60 0x7f8320032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.305+0000 7f8326ffd700 1 -- 172.21.15.47:0/2331071721 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8334044480 con 0x7f83380fab80 2023-11-15T04:28:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.305+0000 7f833da68700 1 --2- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f832002fe60 0x7f8320032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.305+0000 7f833da68700 1 --2- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f832002fe60 0x7f8320032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f83280096a0 tx=0x7f8328006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.309+0000 7f8326ffd700 1 -- 172.21.15.47:0/2331071721 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8334022960 con 0x7f83380fab80 2023-11-15T04:28:55.505 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.504+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 --> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f8338102780 con 0x7f832002fe60 2023-11-15T04:28:55.509 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.508+0000 7f8326ffd700 1 -- 172.21.15.47:0/2331071721 <== mgr.14144 v2:172.21.15.47:6800/826374076 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f8338102780 con 0x7f832002fe60 2023-11-15T04:28:55.509 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-15T04:28:55.512 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.510+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f832002fe60 msgr2=0x7f8320032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:55.512 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.510+0000 7f83404cd700 1 --2- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f832002fe60 0x7f8320032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f83280096a0 tx=0x7f8328006b40).stop 2023-11-15T04:28:55.512 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.510+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 msgr2=0x7f8338108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:55.513 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.510+0000 7f83404cd700 1 --2- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f8338108770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8334006800 tx=0x7f8334006f70).stop 2023-11-15T04:28:55.513 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.510+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 shutdown_connections 2023-11-15T04:28:55.513 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.510+0000 7f83404cd700 1 --2- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f832002fe60 0x7f8320032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:55.513 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.510+0000 7f83404cd700 1 --2- 172.21.15.47:0/2331071721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83380fab80 0x7f8338108770 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:55.513 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.510+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 >> 172.21.15.47:0/2331071721 conn(0x7f83380f63c0 msgr2=0x7f83380ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:55.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.511+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 shutdown_connections 2023-11-15T04:28:55.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:55.511+0000 7f83404cd700 1 -- 172.21.15.47:0/2331071721 wait complete. 2023-11-15T04:28:55.810 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:55 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:54.586885+0000 mgr.smithi047.zgysuz (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:55.811 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:55 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:28:54.587425+0000 mgr.smithi047.zgysuz (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-15T04:28:55.811 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:55 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:54.687741+0000 mon.smithi047 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi047.zgysuz(active, since 2s) 2023-11-15T04:28:55.811 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:55 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:55.509473+0000 mon.smithi047 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying alertmanager service with default placement... 2023-11-15T04:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-15T04:28:56.248 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.246+0000 7fdd4ec28700 1 Processor -- start 2023-11-15T04:28:56.248 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.246+0000 7fdd4ec28700 1 -- start start 2023-11-15T04:28:56.248 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.246+0000 7fdd4ec28700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd480fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:56.248 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.246+0000 7fdd4ec28700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480fc5c0 con 0x7fdd480fbcb0 2023-11-15T04:28:56.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.246+0000 7fdd4c9c4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd480fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:56.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.246+0000 7fdd4c9c4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd480fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57396/0 (socket says 172.21.15.47:57396) 2023-11-15T04:28:56.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.246+0000 7fdd4c9c4700 1 -- 172.21.15.47:0/3398858261 learned_addr learned my addr 172.21.15.47:0/3398858261 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:56.249 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.246+0000 7fdd4c9c4700 1 -- 172.21.15.47:0/3398858261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480fc700 con 0x7fdd480fbcb0 2023-11-15T04:28:56.253 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.247+0000 7fdd4c9c4700 1 --2- 172.21.15.47:0/3398858261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd480fc0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fdd3c00b5f0 tx=0x7fdd3c006ee0).ready entity=mon.0 client_cookie=9e6aa4b95d20f829 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.247+0000 7fdd477fe700 1 -- 172.21.15.47:0/3398858261 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3c00d600 con 0x7fdd480fbcb0 2023-11-15T04:28:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.247+0000 7fdd477fe700 1 -- 172.21.15.47:0/3398858261 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdd3c00dbd0 con 0x7fdd480fbcb0 2023-11-15T04:28:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.247+0000 7fdd477fe700 1 -- 172.21.15.47:0/3398858261 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3c011c00 con 0x7fdd480fbcb0 2023-11-15T04:28:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.248+0000 7fdd4ec28700 1 -- 172.21.15.47:0/3398858261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 msgr2=0x7fdd480fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.248+0000 7fdd4ec28700 1 --2- 172.21.15.47:0/3398858261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd480fc0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fdd3c00b5f0 tx=0x7fdd3c006ee0).stop 2023-11-15T04:28:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.248+0000 7fdd4ec28700 1 -- 172.21.15.47:0/3398858261 shutdown_connections 2023-11-15T04:28:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.248+0000 7fdd4ec28700 1 --2- 172.21.15.47:0/3398858261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd480fc0d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.248+0000 7fdd4ec28700 1 -- 172.21.15.47:0/3398858261 >> 172.21.15.47:0/3398858261 conn(0x7fdd480f74b0 msgr2=0x7fdd480f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.248+0000 7fdd4ec28700 1 -- 172.21.15.47:0/3398858261 shutdown_connections 2023-11-15T04:28:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.248+0000 7fdd4ec28700 1 -- 172.21.15.47:0/3398858261 wait complete. 2023-11-15T04:28:56.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.249+0000 7fdd4ec28700 1 Processor -- start 2023-11-15T04:28:56.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.249+0000 7fdd4ec28700 1 -- start start 2023-11-15T04:28:56.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.249+0000 7fdd4ec28700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd4810aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:56.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.249+0000 7fdd4ec28700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4810af00 con 0x7fdd480fbcb0 2023-11-15T04:28:56.256 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.249+0000 7fdd4c9c4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd4810aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.249+0000 7fdd4c9c4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd4810aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57400/0 (socket says 172.21.15.47:57400) 2023-11-15T04:28:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.249+0000 7fdd4c9c4700 1 -- 172.21.15.47:0/1450180182 learned_addr learned my addr 172.21.15.47:0/1450180182 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.250+0000 7fdd4c9c4700 1 -- 172.21.15.47:0/1450180182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c00b200 con 0x7fdd480fbcb0 2023-11-15T04:28:56.258 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.250+0000 7fdd4c9c4700 1 --2- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd4810aa10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fdd3c00b1d0 tx=0x7fdd3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:56.258 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.250+0000 7fdd457fa700 1 -- 172.21.15.47:0/1450180182 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3c01b410 con 0x7fdd480fbcb0 2023-11-15T04:28:56.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.250+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4810b100 con 0x7fdd480fbcb0 2023-11-15T04:28:56.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.250+0000 7fdd457fa700 1 -- 172.21.15.47:0/1450180182 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdd3c01b9e0 con 0x7fdd480fbcb0 2023-11-15T04:28:56.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.250+0000 7fdd457fa700 1 -- 172.21.15.47:0/1450180182 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3c01a840 con 0x7fdd480fbcb0 2023-11-15T04:28:56.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.250+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4810b510 con 0x7fdd480fbcb0 2023-11-15T04:28:56.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.251+0000 7fdd457fa700 1 -- 172.21.15.47:0/1450180182 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fdd3c00f070 con 0x7fdd480fbcb0 2023-11-15T04:28:56.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.251+0000 7fdd457fa700 1 --2- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fdd2c02fe60 0x7fdd2c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:56.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.251+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd48045ad0 con 0x7fdd480fbcb0 2023-11-15T04:28:56.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.251+0000 7fdd457fa700 1 -- 172.21.15.47:0/1450180182 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fdd3c041520 con 0x7fdd480fbcb0 2023-11-15T04:28:56.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.251+0000 7fdd47fff700 1 --2- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fdd2c02fe60 0x7fdd2c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:56.262 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.252+0000 7fdd47fff700 1 --2- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fdd2c02fe60 0x7fdd2c032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdd340096a0 tx=0x7fdd34006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:56.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.256+0000 7fdd457fa700 1 -- 172.21.15.47:0/1450180182 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd3c025460 con 0x7fdd480fbcb0 2023-11-15T04:28:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.450+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 --> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd4810bdf0 con 0x7fdd2c02fe60 2023-11-15T04:28:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.454+0000 7fdd457fa700 1 -- 172.21.15.47:0/1450180182 <== mgr.14144 v2:172.21.15.47:6800/826374076 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fdd4810bdf0 con 0x7fdd2c02fe60 2023-11-15T04:28:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-15T04:28:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.456+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fdd2c02fe60 msgr2=0x7fdd2c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.456+0000 7fdd4ec28700 1 --2- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fdd2c02fe60 0x7fdd2c032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdd340096a0 tx=0x7fdd34006b40).stop 2023-11-15T04:28:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.456+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 msgr2=0x7fdd4810aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.456+0000 7fdd4ec28700 1 --2- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd4810aa10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fdd3c00b1d0 tx=0x7fdd3c006ee0).stop 2023-11-15T04:28:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.456+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 shutdown_connections 2023-11-15T04:28:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.456+0000 7fdd4ec28700 1 --2- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fdd2c02fe60 0x7fdd2c032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.456+0000 7fdd4ec28700 1 --2- 172.21.15.47:0/1450180182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd480fbcb0 0x7fdd4810aa10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.456+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 >> 172.21.15.47:0/1450180182 conn(0x7fdd480f74b0 msgr2=0x7fdd480f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:56.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.457+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 shutdown_connections 2023-11-15T04:28:56.460 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:56.457+0000 7fdd4ec28700 1 -- 172.21.15.47:0/1450180182 wait complete. 2023-11-15T04:28:56.735 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:55.506383+0000 mgr.smithi047.zgysuz (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:56.735 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:28:55.507124+0000 mgr.smithi047.zgysuz (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-15T04:28:56.735 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:56.455271+0000 mon.smithi047 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.47:0/1462099771' entity='mgr.smithi047.zgysuz' 2023-11-15T04:28:56.790 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-15T04:28:57.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.169+0000 7f46b9069700 1 Processor -- start 2023-11-15T04:28:57.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.169+0000 7f46b9069700 1 -- start start 2023-11-15T04:28:57.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.170+0000 7f46b9069700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b4100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:57.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.170+0000 7f46b9069700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40fc750 con 0x7f46b40703b0 2023-11-15T04:28:57.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.170+0000 7f46b2d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b4100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:57.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.170+0000 7f46b2d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b4100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57406/0 (socket says 172.21.15.47:57406) 2023-11-15T04:28:57.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.170+0000 7f46b2d9d700 1 -- 172.21.15.47:0/3322205560 learned_addr learned my addr 172.21.15.47:0/3322205560 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:57.175 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.170+0000 7f46b2d9d700 1 -- 172.21.15.47:0/3322205560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b4101250 con 0x7f46b40703b0 2023-11-15T04:28:57.175 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.170+0000 7f46b2d9d700 1 --2- 172.21.15.47:0/3322205560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b4100d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f46a400b5f0 tx=0x7f46a4006ee0).ready entity=mon.0 client_cookie=b5f2e687e433106 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:57.175 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b259c700 1 -- 172.21.15.47:0/3322205560 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46a400d600 con 0x7f46b40703b0 2023-11-15T04:28:57.175 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b259c700 1 -- 172.21.15.47:0/3322205560 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f46a400dbd0 con 0x7f46b40703b0 2023-11-15T04:28:57.176 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b259c700 1 -- 172.21.15.47:0/3322205560 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46a4011c00 con 0x7f46b40703b0 2023-11-15T04:28:57.176 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b9069700 1 -- 172.21.15.47:0/3322205560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 msgr2=0x7f46b4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:57.176 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b9069700 1 --2- 172.21.15.47:0/3322205560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b4100d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f46a400b5f0 tx=0x7f46a4006ee0).stop 2023-11-15T04:28:57.176 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b9069700 1 -- 172.21.15.47:0/3322205560 shutdown_connections 2023-11-15T04:28:57.176 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b9069700 1 --2- 172.21.15.47:0/3322205560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b4100d60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:57.177 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b9069700 1 -- 172.21.15.47:0/3322205560 >> 172.21.15.47:0/3322205560 conn(0x7f46b40f63c0 msgr2=0x7f46b40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:57.177 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b9069700 1 -- 172.21.15.47:0/3322205560 shutdown_connections 2023-11-15T04:28:57.177 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.171+0000 7f46b9069700 1 -- 172.21.15.47:0/3322205560 wait complete. 2023-11-15T04:28:57.177 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.172+0000 7f46b9069700 1 Processor -- start 2023-11-15T04:28:57.178 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.172+0000 7f46b9069700 1 -- start start 2023-11-15T04:28:57.180 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46b9069700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b410cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:57.180 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46b9069700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b410d1d0 con 0x7f46b40703b0 2023-11-15T04:28:57.180 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46b2d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b410cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:57.180 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46b2d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b410cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57414/0 (socket says 172.21.15.47:57414) 2023-11-15T04:28:57.181 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46b2d9d700 1 -- 172.21.15.47:0/3161623436 learned_addr learned my addr 172.21.15.47:0/3161623436 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:57.181 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46b2d9d700 1 -- 172.21.15.47:0/3161623436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a400b200 con 0x7f46b40703b0 2023-11-15T04:28:57.181 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46b2d9d700 1 --2- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b410cce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f46a400e040 tx=0x7f46a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:57.181 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46abfff700 1 -- 172.21.15.47:0/3161623436 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46a400d9f0 con 0x7f46b40703b0 2023-11-15T04:28:57.181 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46abfff700 1 -- 172.21.15.47:0/3161623436 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f46a401a9f0 con 0x7f46b40703b0 2023-11-15T04:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b410d3d0 con 0x7f46b40703b0 2023-11-15T04:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.173+0000 7f46abfff700 1 -- 172.21.15.47:0/3161623436 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46a401b8c0 con 0x7f46b40703b0 2023-11-15T04:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.174+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b410d7e0 con 0x7f46b40703b0 2023-11-15T04:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.174+0000 7f46abfff700 1 -- 172.21.15.47:0/3161623436 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f46a400f070 con 0x7f46b40703b0 2023-11-15T04:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.175+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b4045ad0 con 0x7f46b40703b0 2023-11-15T04:28:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.175+0000 7f46abfff700 1 --2- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f469402feb0 0x7f4694032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.175+0000 7f46abfff700 1 -- 172.21.15.47:0/3161623436 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f46a4041430 con 0x7f46b40703b0 2023-11-15T04:28:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.175+0000 7f46aa59c700 1 --2- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f469402feb0 0x7f4694032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.175+0000 7f46aa59c700 1 --2- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f469402feb0 0x7f4694032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f469c0096a0 tx=0x7f469c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:57.185 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.180+0000 7f46abfff700 1 -- 172.21.15.47:0/3161623436 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46a40257e0 con 0x7f46b40703b0 2023-11-15T04:28:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.348+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f46b4101db0 con 0x7f46b40703b0 2023-11-15T04:28:57.351 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.351+0000 7f46abfff700 1 -- 172.21.15.47:0/3161623436 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f46a4016030 con 0x7f46b40703b0 2023-11-15T04:28:57.355 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.353+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f469402feb0 msgr2=0x7f4694032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:57.355 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.353+0000 7f46b9069700 1 --2- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f469402feb0 0x7f4694032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f469c0096a0 tx=0x7f469c006b40).stop 2023-11-15T04:28:57.355 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.353+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 msgr2=0x7f46b410cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:57.355 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.353+0000 7f46b9069700 1 --2- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b410cce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f46a400e040 tx=0x7f46a4006ee0).stop 2023-11-15T04:28:57.356 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.354+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 shutdown_connections 2023-11-15T04:28:57.356 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.354+0000 7f46b9069700 1 --2- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f469402feb0 0x7f4694032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:57.356 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.354+0000 7f46b9069700 1 --2- 172.21.15.47:0/3161623436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46b40703b0 0x7f46b410cce0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:57.356 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.354+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 >> 172.21.15.47:0/3161623436 conn(0x7f46b40f63c0 msgr2=0x7f46b40f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:57.357 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.355+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 shutdown_connections 2023-11-15T04:28:57.357 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:57.355+0000 7f46b9069700 1 -- 172.21.15.47:0/3161623436 wait complete. 2023-11-15T04:28:57.716 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-15T04:28:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:56.452296+0000 mgr.smithi047.zgysuz (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:28:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:28:56.452873+0000 mgr.smithi047.zgysuz (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-15T04:28:58.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:57.352219+0000 mon.smithi047 (mon.0) 83 : audit [INF] from='client.? 172.21.15.47:0/3161623436' entity='client.admin' 2023-11-15T04:28:58.103 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.094+0000 7fd1dfe24700 1 Processor -- start 2023-11-15T04:28:58.104 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.094+0000 7fd1dfe24700 1 -- start start 2023-11-15T04:28:58.104 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.094+0000 7fd1dfe24700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d8100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:58.105 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.094+0000 7fd1dfe24700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80fc750 con 0x7fd1d80703b0 2023-11-15T04:28:58.105 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.094+0000 7fd1ddbc0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d8100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:58.106 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.094+0000 7fd1ddbc0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d8100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57418/0 (socket says 172.21.15.47:57418) 2023-11-15T04:28:58.106 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.094+0000 7fd1ddbc0700 1 -- 172.21.15.47:0/1264510104 learned_addr learned my addr 172.21.15.47:0/1264510104 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:58.107 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.095+0000 7fd1ddbc0700 1 -- 172.21.15.47:0/1264510104 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8101250 con 0x7fd1d80703b0 2023-11-15T04:28:58.107 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.095+0000 7fd1ddbc0700 1 --2- 172.21.15.47:0/1264510104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d8100d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd1d400b5f0 tx=0x7fd1d4006ee0).ready entity=mon.0 client_cookie=74cacc70093e286b server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:58.107 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.095+0000 7fd1dcbbe700 1 -- 172.21.15.47:0/1264510104 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d400d600 con 0x7fd1d80703b0 2023-11-15T04:28:58.108 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.095+0000 7fd1dcbbe700 1 -- 172.21.15.47:0/1264510104 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd1d400dbd0 con 0x7fd1d80703b0 2023-11-15T04:28:58.108 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.095+0000 7fd1dcbbe700 1 -- 172.21.15.47:0/1264510104 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d4011c00 con 0x7fd1d80703b0 2023-11-15T04:28:58.109 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.096+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1264510104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 msgr2=0x7fd1d8100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:58.109 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.096+0000 7fd1dfe24700 1 --2- 172.21.15.47:0/1264510104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d8100d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd1d400b5f0 tx=0x7fd1d4006ee0).stop 2023-11-15T04:28:58.109 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.096+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1264510104 shutdown_connections 2023-11-15T04:28:58.110 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.096+0000 7fd1dfe24700 1 --2- 172.21.15.47:0/1264510104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d8100d60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:58.110 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.096+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1264510104 >> 172.21.15.47:0/1264510104 conn(0x7fd1d80f63c0 msgr2=0x7fd1d80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:58.110 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.096+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1264510104 shutdown_connections 2023-11-15T04:28:58.111 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.096+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1264510104 wait complete. 2023-11-15T04:28:58.111 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.097+0000 7fd1dfe24700 1 Processor -- start 2023-11-15T04:28:58.111 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.097+0000 7fd1dfe24700 1 -- start start 2023-11-15T04:28:58.112 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.097+0000 7fd1dfe24700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d810cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:58.112 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.097+0000 7fd1dfe24700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d810d0f0 con 0x7fd1d80703b0 2023-11-15T04:28:58.112 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.097+0000 7fd1ddbc0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d810cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:58.112 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.097+0000 7fd1ddbc0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d810cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57422/0 (socket says 172.21.15.47:57422) 2023-11-15T04:28:58.112 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.097+0000 7fd1ddbc0700 1 -- 172.21.15.47:0/1565356215 learned_addr learned my addr 172.21.15.47:0/1565356215 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:58.113 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.097+0000 7fd1ddbc0700 1 -- 172.21.15.47:0/1565356215 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d400b200 con 0x7fd1d80703b0 2023-11-15T04:28:58.113 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.097+0000 7fd1ddbc0700 1 --2- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d810cc00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd1d400b190 tx=0x7fd1d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:58.113 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.098+0000 7fd1c67fc700 1 -- 172.21.15.47:0/1565356215 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d401b410 con 0x7fd1d80703b0 2023-11-15T04:28:58.114 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.098+0000 7fd1c67fc700 1 -- 172.21.15.47:0/1565356215 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd1d401b9e0 con 0x7fd1d80703b0 2023-11-15T04:28:58.114 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.098+0000 7fd1c67fc700 1 -- 172.21.15.47:0/1565356215 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d401a880 con 0x7fd1d80703b0 2023-11-15T04:28:58.115 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.098+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d810d2f0 con 0x7fd1d80703b0 2023-11-15T04:28:58.115 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.098+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d810d700 con 0x7fd1d80703b0 2023-11-15T04:28:58.115 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.099+0000 7fd1c67fc700 1 -- 172.21.15.47:0/1565356215 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd1d400f070 con 0x7fd1d80703b0 2023-11-15T04:28:58.116 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.099+0000 7fd1c67fc700 1 --2- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fd1c002fe60 0x7fd1c0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:58.116 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.099+0000 7fd1c67fc700 1 -- 172.21.15.47:0/1565356215 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd1d4041490 con 0x7fd1d80703b0 2023-11-15T04:28:58.116 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.099+0000 7fd1dd3bf700 1 --2- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fd1c002fe60 0x7fd1c0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:58.117 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.099+0000 7fd1dd3bf700 1 --2- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fd1c002fe60 0x7fd1c0032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd1c80096a0 tx=0x7fd1c8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:58.117 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.099+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d8059100 con 0x7fd1d80703b0 2023-11-15T04:28:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.104+0000 7fd1c67fc700 1 -- 172.21.15.47:0/1565356215 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1d4016070 con 0x7fd1d80703b0 2023-11-15T04:28:58.269 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.268+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd1d8045ad0 con 0x7fd1d80703b0 2023-11-15T04:28:58.271 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.271+0000 7fd1c67fc700 1 -- 172.21.15.47:0/1565356215 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd1d40224f0 con 0x7fd1d80703b0 2023-11-15T04:28:58.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fd1c002fe60 msgr2=0x7fd1c0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:58.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 --2- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fd1c002fe60 0x7fd1c0032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd1c80096a0 tx=0x7fd1c8006b40).stop 2023-11-15T04:28:58.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 msgr2=0x7fd1d810cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:58.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 --2- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d810cc00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd1d400b190 tx=0x7fd1d4006ee0).stop 2023-11-15T04:28:58.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 shutdown_connections 2023-11-15T04:28:58.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 --2- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7fd1c002fe60 0x7fd1c0032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:58.276 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 --2- 172.21.15.47:0/1565356215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d80703b0 0x7fd1d810cc00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:58.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 >> 172.21.15.47:0/1565356215 conn(0x7fd1d80f63c0 msgr2=0x7fd1d80f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:58.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 shutdown_connections 2023-11-15T04:28:58.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:58.273+0000 7fd1dfe24700 1 -- 172.21.15.47:0/1565356215 wait complete. 2023-11-15T04:28:58.623 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling the dashboard module... 2023-11-15T04:28:58.623 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-15T04:28:59.016 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.013+0000 7ff967fff700 1 Processor -- start 2023-11-15T04:28:59.016 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.013+0000 7ff967fff700 1 -- start start 2023-11-15T04:28:59.016 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.013+0000 7ff967fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff9680fe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:59.017 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.013+0000 7ff967fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9680fec40 con 0x7ff968070430 2023-11-15T04:28:59.017 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.014+0000 7ff966ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff9680fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:59.017 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.014+0000 7ff966ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff9680fe750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57434/0 (socket says 172.21.15.47:57434) 2023-11-15T04:28:59.017 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.014+0000 7ff966ffd700 1 -- 172.21.15.47:0/4192892126 learned_addr learned my addr 172.21.15.47:0/4192892126 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:59.017 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.014+0000 7ff966ffd700 1 -- 172.21.15.47:0/4192892126 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9680fed80 con 0x7ff968070430 2023-11-15T04:28:59.018 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.014+0000 7ff966ffd700 1 --2- 172.21.15.47:0/4192892126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff9680fe750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7ff95000b5f0 tx=0x7ff950006ee0).ready entity=mon.0 client_cookie=b1a5a8289ef2189e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:59.018 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.014+0000 7ff965ffb700 1 -- 172.21.15.47:0/4192892126 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff95000d600 con 0x7ff968070430 2023-11-15T04:28:59.018 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.015+0000 7ff965ffb700 1 -- 172.21.15.47:0/4192892126 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff95000dbd0 con 0x7ff968070430 2023-11-15T04:28:59.018 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.015+0000 7ff965ffb700 1 -- 172.21.15.47:0/4192892126 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff950011c00 con 0x7ff968070430 2023-11-15T04:28:59.020 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.015+0000 7ff967fff700 1 -- 172.21.15.47:0/4192892126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 msgr2=0x7ff9680fe750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.015+0000 7ff967fff700 1 --2- 172.21.15.47:0/4192892126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff9680fe750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7ff95000b5f0 tx=0x7ff950006ee0).stop 2023-11-15T04:28:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.015+0000 7ff967fff700 1 -- 172.21.15.47:0/4192892126 shutdown_connections 2023-11-15T04:28:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.015+0000 7ff967fff700 1 --2- 172.21.15.47:0/4192892126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff9680fe750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.015+0000 7ff967fff700 1 -- 172.21.15.47:0/4192892126 >> 172.21.15.47:0/4192892126 conn(0x7ff9680f6210 msgr2=0x7ff9680f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:59.022 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.015+0000 7ff967fff700 1 -- 172.21.15.47:0/4192892126 shutdown_connections 2023-11-15T04:28:59.022 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.015+0000 7ff967fff700 1 -- 172.21.15.47:0/4192892126 wait complete. 2023-11-15T04:28:59.022 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.016+0000 7ff967fff700 1 Processor -- start 2023-11-15T04:28:59.022 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.016+0000 7ff967fff700 1 -- start start 2023-11-15T04:28:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.016+0000 7ff967fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff96810a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.016+0000 7ff967fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96810ad60 con 0x7ff968070430 2023-11-15T04:28:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.016+0000 7ff966ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff96810a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.016+0000 7ff966ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff96810a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57438/0 (socket says 172.21.15.47:57438) 2023-11-15T04:28:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.016+0000 7ff966ffd700 1 -- 172.21.15.47:0/1038500149 learned_addr learned my addr 172.21.15.47:0/1038500149 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:28:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.017+0000 7ff966ffd700 1 -- 172.21.15.47:0/1038500149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95000b200 con 0x7ff968070430 2023-11-15T04:28:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.017+0000 7ff966ffd700 1 --2- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff96810a870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff95000b190 tx=0x7ff950006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.017+0000 7ff95f7fe700 1 -- 172.21.15.47:0/1038500149 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff95001b410 con 0x7ff968070430 2023-11-15T04:28:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.017+0000 7ff95f7fe700 1 -- 172.21.15.47:0/1038500149 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff95001b9e0 con 0x7ff968070430 2023-11-15T04:28:59.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.017+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96810af60 con 0x7ff968070430 2023-11-15T04:28:59.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.017+0000 7ff95f7fe700 1 -- 172.21.15.47:0/1038500149 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff95001a880 con 0x7ff968070430 2023-11-15T04:28:59.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.017+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96810b370 con 0x7ff968070430 2023-11-15T04:28:59.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.018+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff968045ad0 con 0x7ff968070430 2023-11-15T04:28:59.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.019+0000 7ff95f7fe700 1 -- 172.21.15.47:0/1038500149 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff95000f070 con 0x7ff968070430 2023-11-15T04:28:59.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.019+0000 7ff95f7fe700 1 --2- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7ff94802fe60 0x7ff948032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:28:59.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.019+0000 7ff95f7fe700 1 -- 172.21.15.47:0/1038500149 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff950041e90 con 0x7ff968070430 2023-11-15T04:28:59.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.019+0000 7ff9667fc700 1 --2- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7ff94802fe60 0x7ff948032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:28:59.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.020+0000 7ff9667fc700 1 --2- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7ff94802fe60 0x7ff948032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff9580096a0 tx=0x7ff958006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:28:59.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.023+0000 7ff95f7fe700 1 -- 172.21.15.47:0/1038500149 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff95002db70 con 0x7ff968070430 2023-11-15T04:28:59.231 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.229+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff9680ffa00 con 0x7ff968070430 2023-11-15T04:28:59.274 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.273+0000 7ff95f7fe700 1 -- 172.21.15.47:0/1038500149 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff95001bc10 con 0x7ff968070430 2023-11-15T04:28:59.275 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.274+0000 7ff95f7fe700 1 -- 172.21.15.47:0/1038500149 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff9500182e0 con 0x7ff968070430 2023-11-15T04:28:59.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7ff94802fe60 msgr2=0x7ff948032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:59.278 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 --2- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7ff94802fe60 0x7ff948032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff9580096a0 tx=0x7ff958006b40).stop 2023-11-15T04:28:59.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 msgr2=0x7ff96810a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:28:59.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 --2- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff96810a870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff95000b190 tx=0x7ff950006ee0).stop 2023-11-15T04:28:59.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 shutdown_connections 2023-11-15T04:28:59.279 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 --2- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7ff94802fe60 0x7ff948032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:59.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 --2- 172.21.15.47:0/1038500149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff968070430 0x7ff96810a870 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:28:59.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 >> 172.21.15.47:0/1038500149 conn(0x7ff9680f6210 msgr2=0x7ff9680f7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:28:59.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 shutdown_connections 2023-11-15T04:28:59.280 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:28:59.276+0000 7ff967fff700 1 -- 172.21.15.47:0/1038500149 wait complete. 2023-11-15T04:28:59.600 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:59 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:58.272245+0000 mon.smithi047 (mon.0) 84 : audit [INF] from='client.? 172.21.15.47:0/1565356215' entity='client.admin' 2023-11-15T04:28:59.601 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:28:59 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:59.230527+0000 mon.smithi047 (mon.0) 85 : audit [INF] from='client.? 172.21.15.47:0/1038500149' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-15T04:28:59.657 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T04:29:00.067 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.065+0000 7f6995c81700 1 Processor -- start 2023-11-15T04:29:00.067 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.065+0000 7f6995c81700 1 -- start start 2023-11-15T04:29:00.067 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.066+0000 7f6995c81700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f69900fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:00.068 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.066+0000 7f6995c81700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69900fb4a0 con 0x7f69900fab90 2023-11-15T04:29:00.068 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.066+0000 7f698f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f69900fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:00.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.066+0000 7f698f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f69900fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57468/0 (socket says 172.21.15.47:57468) 2023-11-15T04:29:00.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.066+0000 7f698f7fe700 1 -- 172.21.15.47:0/2549979269 learned_addr learned my addr 172.21.15.47:0/2549979269 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:00.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.066+0000 7f698f7fe700 1 -- 172.21.15.47:0/2549979269 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69900fb5e0 con 0x7f69900fab90 2023-11-15T04:29:00.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.066+0000 7f698f7fe700 1 --2- 172.21.15.47:0/2549979269 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f69900fafb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f698000b870 tx=0x7f6980006f70).ready entity=mon.0 client_cookie=9191b1d5221fc76e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:00.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.067+0000 7f698e7fc700 1 -- 172.21.15.47:0/2549979269 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f698000da20 con 0x7f69900fab90 2023-11-15T04:29:00.070 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.067+0000 7f698e7fc700 1 -- 172.21.15.47:0/2549979269 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6980010470 con 0x7f69900fab90 2023-11-15T04:29:00.070 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.067+0000 7f698e7fc700 1 -- 172.21.15.47:0/2549979269 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f698000f5b0 con 0x7f69900fab90 2023-11-15T04:29:00.071 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.067+0000 7f6995c81700 1 -- 172.21.15.47:0/2549979269 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 msgr2=0x7f69900fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:00.071 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.067+0000 7f6995c81700 1 --2- 172.21.15.47:0/2549979269 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f69900fafb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f698000b870 tx=0x7f6980006f70).stop 2023-11-15T04:29:00.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.068+0000 7f6995c81700 1 -- 172.21.15.47:0/2549979269 shutdown_connections 2023-11-15T04:29:00.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.068+0000 7f6995c81700 1 --2- 172.21.15.47:0/2549979269 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f69900fafb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:00.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.068+0000 7f6995c81700 1 -- 172.21.15.47:0/2549979269 >> 172.21.15.47:0/2549979269 conn(0x7f69900f63b0 msgr2=0x7f69900f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:00.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.068+0000 7f6995c81700 1 -- 172.21.15.47:0/2549979269 shutdown_connections 2023-11-15T04:29:00.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.068+0000 7f6995c81700 1 -- 172.21.15.47:0/2549979269 wait complete. 2023-11-15T04:29:00.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.068+0000 7f6995c81700 1 Processor -- start 2023-11-15T04:29:00.073 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.069+0000 7f6995c81700 1 -- start start 2023-11-15T04:29:00.073 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.069+0000 7f6995c81700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f6990108730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:00.074 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.069+0000 7f6995c81700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6990108c20 con 0x7f69900fab90 2023-11-15T04:29:00.074 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.069+0000 7f698f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f6990108730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:00.075 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.069+0000 7f698f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f6990108730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57476/0 (socket says 172.21.15.47:57476) 2023-11-15T04:29:00.075 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.069+0000 7f698f7fe700 1 -- 172.21.15.47:0/1834301185 learned_addr learned my addr 172.21.15.47:0/1834301185 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:00.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.069+0000 7f698f7fe700 1 -- 172.21.15.47:0/1834301185 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698000b480 con 0x7f69900fab90 2023-11-15T04:29:00.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.069+0000 7f698f7fe700 1 --2- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f6990108730 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6980006800 tx=0x7f6980006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:00.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.070+0000 7f6977fff700 1 -- 172.21.15.47:0/1834301185 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f698000f820 con 0x7f69900fab90 2023-11-15T04:29:00.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.070+0000 7f6977fff700 1 -- 172.21.15.47:0/1834301185 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f698000fdf0 con 0x7f69900fab90 2023-11-15T04:29:00.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.070+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6990108e20 con 0x7f69900fab90 2023-11-15T04:29:00.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.070+0000 7f6977fff700 1 -- 172.21.15.47:0/1834301185 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6980019d50 con 0x7f69900fab90 2023-11-15T04:29:00.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.070+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6990109230 con 0x7f69900fab90 2023-11-15T04:29:00.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.071+0000 7f6977fff700 1 -- 172.21.15.47:0/1834301185 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f6980016030 con 0x7f69900fab90 2023-11-15T04:29:00.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.071+0000 7f6977fff700 1 --2- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f697c02ff00 0x7f697c0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:00.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.071+0000 7f6977fff700 1 -- 172.21.15.47:0/1834301185 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6980045170 con 0x7f69900fab90 2023-11-15T04:29:00.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.071+0000 7f698effd700 1 -- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f697c02ff00 msgr2=0x7f697c0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/826374076 2023-11-15T04:29:00.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.071+0000 7f698effd700 1 --2- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f697c02ff00 0x7f697c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:29:00.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.071+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6990059100 con 0x7f69900fab90 2023-11-15T04:29:00.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.076+0000 7f6977fff700 1 -- 172.21.15.47:0/1834301185 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6980014080 con 0x7f69900fab90 2023-11-15T04:29:00.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.272+0000 7f698effd700 1 -- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f697c02ff00 msgr2=0x7f697c0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/826374076 2023-11-15T04:29:00.273 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.272+0000 7f698effd700 1 --2- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f697c02ff00 0x7f697c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:29:00.327 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.325+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f69900fc8a0 con 0x7f69900fab90 2023-11-15T04:29:00.327 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.326+0000 7f6977fff700 1 -- 172.21.15.47:0/1834301185 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f69800189a0 con 0x7f69900fab90 2023-11-15T04:29:00.327 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2023-11-15T04:29:00.327 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 13, 2023-11-15T04:29:00.328 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": true, 2023-11-15T04:29:00.328 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "active_name": "smithi047.zgysuz", 2023-11-15T04:29:00.328 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T04:29:00.328 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2023-11-15T04:29:00.329 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.327+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f697c02ff00 msgr2=0x7f697c0323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:29:00.330 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.327+0000 7f6995c81700 1 --2- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f697c02ff00 0x7f697c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:29:00.330 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.327+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 msgr2=0x7f6990108730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:00.331 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.327+0000 7f6995c81700 1 --2- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f6990108730 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6980006800 tx=0x7f6980006f70).stop 2023-11-15T04:29:00.331 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.328+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 shutdown_connections 2023-11-15T04:29:00.331 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.328+0000 7f6995c81700 1 --2- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7f697c02ff00 0x7f697c0323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:29:00.331 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.328+0000 7f6995c81700 1 --2- 172.21.15.47:0/1834301185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69900fab90 0x7f6990108730 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:00.331 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.328+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 >> 172.21.15.47:0/1834301185 conn(0x7f69900f63b0 msgr2=0x7f69900ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:00.332 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.328+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 shutdown_connections 2023-11-15T04:29:00.332 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:00.328+0000 7f6995c81700 1 -- 172.21.15.47:0/1834301185 wait complete. 2023-11-15T04:29:00.473 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:28:59.275447+0000 mon.smithi047 (mon.0) 86 : audit [INF] from='client.? 172.21.15.47:0/1038500149' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-15T04:29:00.474 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:28:59.275497+0000 mon.smithi047 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi047.zgysuz(active, since 6s) 2023-11-15T04:29:00.727 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for the mgr to restart... 2023-11-15T04:29:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr epoch 13... 2023-11-15T04:29:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T04:29:01.338 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.337+0000 7feadadba700 1 Processor -- start 2023-11-15T04:29:01.339 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.337+0000 7feadadba700 1 -- start start 2023-11-15T04:29:01.339 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.337+0000 7feadadba700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:01.340 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.337+0000 7feadadba700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead40fedd0 con 0x7fead4070430 2023-11-15T04:29:01.340 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.337+0000 7fead8b56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:01.340 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.337+0000 7fead8b56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57480/0 (socket says 172.21.15.47:57480) 2023-11-15T04:29:01.340 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.337+0000 7fead8b56700 1 -- 172.21.15.47:0/1605305913 learned_addr learned my addr 172.21.15.47:0/1605305913 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:01.343 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.338+0000 7fead8b56700 1 -- 172.21.15.47:0/1605305913 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead40fef10 con 0x7fead4070430 2023-11-15T04:29:01.344 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.338+0000 7fead8b56700 1 --2- 172.21.15.47:0/1605305913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead40fe8e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7feac800b5f0 tx=0x7feac8006ee0).ready entity=mon.0 client_cookie=bb6f819c3d915a31 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:01.344 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.338+0000 7fead37fe700 1 -- 172.21.15.47:0/1605305913 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feac800d600 con 0x7fead4070430 2023-11-15T04:29:01.344 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.338+0000 7fead37fe700 1 -- 172.21.15.47:0/1605305913 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feac800dbd0 con 0x7fead4070430 2023-11-15T04:29:01.344 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.338+0000 7fead37fe700 1 -- 172.21.15.47:0/1605305913 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feac8011c00 con 0x7fead4070430 2023-11-15T04:29:01.344 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.339+0000 7feadadba700 1 -- 172.21.15.47:0/1605305913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 msgr2=0x7fead40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:01.345 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.339+0000 7feadadba700 1 --2- 172.21.15.47:0/1605305913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead40fe8e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7feac800b5f0 tx=0x7feac8006ee0).stop 2023-11-15T04:29:01.345 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.339+0000 7feadadba700 1 -- 172.21.15.47:0/1605305913 shutdown_connections 2023-11-15T04:29:01.345 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.339+0000 7feadadba700 1 --2- 172.21.15.47:0/1605305913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead40fe8e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:01.345 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.339+0000 7feadadba700 1 -- 172.21.15.47:0/1605305913 >> 172.21.15.47:0/1605305913 conn(0x7fead40f63c0 msgr2=0x7fead40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:01.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.339+0000 7feadadba700 1 -- 172.21.15.47:0/1605305913 shutdown_connections 2023-11-15T04:29:01.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.339+0000 7feadadba700 1 -- 172.21.15.47:0/1605305913 wait complete. 2023-11-15T04:29:01.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.340+0000 7feadadba700 1 Processor -- start 2023-11-15T04:29:01.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.340+0000 7feadadba700 1 -- start start 2023-11-15T04:29:01.346 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.340+0000 7feadadba700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:01.347 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.340+0000 7feadadba700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead410aef0 con 0x7fead4070430 2023-11-15T04:29:01.347 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7fead8b56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:01.347 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7fead8b56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57482/0 (socket says 172.21.15.47:57482) 2023-11-15T04:29:01.347 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7fead8b56700 1 -- 172.21.15.47:0/2172585036 learned_addr learned my addr 172.21.15.47:0/2172585036 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:01.347 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7fead8b56700 1 -- 172.21.15.47:0/2172585036 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac800b200 con 0x7fead4070430 2023-11-15T04:29:01.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7fead8b56700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead410aa00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7feac800b190 tx=0x7feac8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:01.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feac801b410 con 0x7fead4070430 2023-11-15T04:29:01.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feac801b9e0 con 0x7fead4070430 2023-11-15T04:29:01.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead410b0f0 con 0x7fead4070430 2023-11-15T04:29:01.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feac801a880 con 0x7fead4070430 2023-11-15T04:29:01.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.341+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead410b500 con 0x7fead4070430 2023-11-15T04:29:01.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.342+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7feac800f070 con 0x7fead4070430 2023-11-15T04:29:01.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.342+0000 7fead17fa700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 0x7feab8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:01.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.342+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 --> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fead4059100 con 0x7feab802feb0 2023-11-15T04:29:01.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.343+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7feac8041490 con 0x7fead4070430 2023-11-15T04:29:01.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.343+0000 7fead3fff700 1 -- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 msgr2=0x7feab8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/826374076 2023-11-15T04:29:01.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.343+0000 7fead3fff700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 0x7feab8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:29:01.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:01 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:00.326917+0000 mon.smithi047 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.47:0/1834301185' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T04:29:01.545 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.544+0000 7fead3fff700 1 -- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 msgr2=0x7feab8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/826374076 2023-11-15T04:29:01.545 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.544+0000 7fead3fff700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 0x7feab8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:29:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.945+0000 7fead3fff700 1 -- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 msgr2=0x7feab8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/826374076 2023-11-15T04:29:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:01.945+0000 7fead3fff700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 0x7feab8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T04:29:02.747 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:02.746+0000 7fead3fff700 1 -- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 msgr2=0x7feab8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/826374076 2023-11-15T04:29:02.747 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:02.746+0000 7fead3fff700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 0x7feab8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T04:29:04.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:04.347+0000 7fead3fff700 1 -- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 msgr2=0x7feab8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/826374076 2023-11-15T04:29:04.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:04.347+0000 7fead3fff700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 0x7feab8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T04:29:05.050 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:05.049+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7feac8025980 con 0x7fead4070430 2023-11-15T04:29:05.051 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:05.049+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 msgr2=0x7feab8032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:29:05.051 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:05.049+0000 7fead17fa700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 0x7feab8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:29:06.053 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.051+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7feac8025460 con 0x7fead4070430 2023-11-15T04:29:06.054 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.051+0000 7fead17fa700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feab8033990 0x7feab8035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:06.054 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.051+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 --> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fead4059100 con 0x7feab8033990 2023-11-15T04:29:06.054 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.051+0000 7fead3fff700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feab8033990 0x7feab8035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:06.055 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.052+0000 7fead3fff700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feab8033990 0x7feab8035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7feac0006fd0 tx=0x7feac0006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:06.055 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.053+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 <== mgr.14164 v2:172.21.15.47:6800/1025769174 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fead4059100 con 0x7feab8033990 2023-11-15T04:29:06.060 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.060+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 --> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fead4045ad0 con 0x7feab8033990 2023-11-15T04:29:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.060+0000 7fead17fa700 1 -- 172.21.15.47:0/2172585036 <== mgr.14164 v2:172.21.15.47:6800/1025769174 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fead4045ad0 con 0x7feab8033990 2023-11-15T04:29:06.063 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2023-11-15T04:29:06.063 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-11-15T04:29:06.063 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T04:29:06.063 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2023-11-15T04:29:06.063 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.060+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feab8033990 msgr2=0x7feab8035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:06.064 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.060+0000 7feadadba700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feab8033990 0x7feab8035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7feac0006fd0 tx=0x7feac0006cf0).stop 2023-11-15T04:29:06.064 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.060+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 msgr2=0x7fead410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:06.064 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.060+0000 7feadadba700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead410aa00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7feac800b190 tx=0x7feac8006ee0).stop 2023-11-15T04:29:06.064 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.061+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 shutdown_connections 2023-11-15T04:29:06.064 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.061+0000 7feadadba700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feab8033990 0x7feab8035d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.061+0000 7feadadba700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fead4070430 0x7fead410aa00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.061+0000 7feadadba700 1 --2- 172.21.15.47:0/2172585036 >> [v2:172.21.15.47:6800/826374076,v1:172.21.15.47:6801/826374076] conn(0x7feab802feb0 0x7feab8032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:29:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.061+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 >> 172.21.15.47:0/2172585036 conn(0x7fead40f63c0 msgr2=0x7fead40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.061+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 shutdown_connections 2023-11-15T04:29:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.061+0000 7feadadba700 1 -- 172.21.15.47:0/2172585036 wait complete. 2023-11-15T04:29:06.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:04.998788+0000 mon.smithi047 (mon.0) 89 : cluster [INF] Active manager daemon smithi047.zgysuz restarted 2023-11-15T04:29:06.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:04.999214+0000 mon.smithi047 (mon.0) 90 : cluster [INF] Activating manager daemon smithi047.zgysuz 2023-11-15T04:29:06.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:05.001103+0000 mon.smithi047 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-15T04:29:06.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:05.051704+0000 mon.smithi047 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi047.zgysuz(active, starting, since 0.0525827s) 2023-11-15T04:29:06.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:05.052959+0000 mon.smithi047 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2023-11-15T04:29:06.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:05.053596+0000 mon.smithi047 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mgr metadata", "who": "smithi047.zgysuz", "id": "smithi047.zgysuz"}]: dispatch 2023-11-15T04:29:06.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:05.054636+0000 mon.smithi047 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:29:06.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:05.054841+0000 mon.smithi047 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:29:06.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:05.055004+0000 mon.smithi047 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:29:06.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:05.068634+0000 mon.smithi047 (mon.0) 98 : cluster [INF] Manager daemon smithi047.zgysuz is now available 2023-11-15T04:29:06.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:05.182364+0000 mon.smithi047 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:29:06.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:05.183990+0000 mon.smithi047 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:29:06.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:05.211504+0000 mon.smithi047 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:29:06.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:05.214542+0000 mon.smithi047 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:29:06.365 INFO:teuthology.orchestra.run.smithi047.stderr:mgr epoch 13 is available 2023-11-15T04:29:06.366 INFO:teuthology.orchestra.run.smithi047.stderr:Generating a dashboard self-signed certificate... 2023-11-15T04:29:06.366 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-15T04:29:06.752 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.749+0000 7fa16d0eb700 1 Processor -- start 2023-11-15T04:29:06.752 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.749+0000 7fa16d0eb700 1 -- start start 2023-11-15T04:29:06.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.750+0000 7fa16d0eb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:06.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.750+0000 7fa16d0eb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1680fedd0 con 0x7fa168070430 2023-11-15T04:29:06.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.750+0000 7fa166d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:06.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.750+0000 7fa166d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39636/0 (socket says 172.21.15.47:39636) 2023-11-15T04:29:06.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.750+0000 7fa166d9d700 1 -- 172.21.15.47:0/2846180009 learned_addr learned my addr 172.21.15.47:0/2846180009 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:06.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.750+0000 7fa166d9d700 1 -- 172.21.15.47:0/2846180009 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1680fef10 con 0x7fa168070430 2023-11-15T04:29:06.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa166d9d700 1 --2- 172.21.15.47:0/2846180009 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fe8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa15800b5f0 tx=0x7fa158006ee0).ready entity=mon.0 client_cookie=8f0d12b80a5def77 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:06.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa165d9b700 1 -- 172.21.15.47:0/2846180009 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa15800d600 con 0x7fa168070430 2023-11-15T04:29:06.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa165d9b700 1 -- 172.21.15.47:0/2846180009 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa15800dbd0 con 0x7fa168070430 2023-11-15T04:29:06.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa165d9b700 1 -- 172.21.15.47:0/2846180009 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa158011c00 con 0x7fa168070430 2023-11-15T04:29:06.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa16d0eb700 1 -- 172.21.15.47:0/2846180009 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 msgr2=0x7fa1680fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:06.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa16d0eb700 1 --2- 172.21.15.47:0/2846180009 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fe8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa15800b5f0 tx=0x7fa158006ee0).stop 2023-11-15T04:29:06.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa16d0eb700 1 -- 172.21.15.47:0/2846180009 shutdown_connections 2023-11-15T04:29:06.756 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa16d0eb700 1 --2- 172.21.15.47:0/2846180009 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fe8e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:06.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa16d0eb700 1 -- 172.21.15.47:0/2846180009 >> 172.21.15.47:0/2846180009 conn(0x7fa1680f63c0 msgr2=0x7fa1680f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:06.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa16d0eb700 1 -- 172.21.15.47:0/2846180009 shutdown_connections 2023-11-15T04:29:06.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.751+0000 7fa16d0eb700 1 -- 172.21.15.47:0/2846180009 wait complete. 2023-11-15T04:29:06.757 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.752+0000 7fa16d0eb700 1 Processor -- start 2023-11-15T04:29:06.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.752+0000 7fa16d0eb700 1 -- start start 2023-11-15T04:29:06.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa16d0eb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:06.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa16d0eb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1680fa7e0 con 0x7fa168070430 2023-11-15T04:29:06.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa166d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:06.758 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa166d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39648/0 (socket says 172.21.15.47:39648) 2023-11-15T04:29:06.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa166d9d700 1 -- 172.21.15.47:0/3657349955 learned_addr learned my addr 172.21.15.47:0/3657349955 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:06.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa166d9d700 1 -- 172.21.15.47:0/3657349955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15800b200 con 0x7fa168070430 2023-11-15T04:29:06.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa166d9d700 1 --2- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fc360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa15800b190 tx=0x7fa158006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:06.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa15f7fe700 1 -- 172.21.15.47:0/3657349955 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa15801b410 con 0x7fa168070430 2023-11-15T04:29:06.759 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1680fa980 con 0x7fa168070430 2023-11-15T04:29:06.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa15f7fe700 1 -- 172.21.15.47:0/3657349955 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa15801b9e0 con 0x7fa168070430 2023-11-15T04:29:06.760 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa15f7fe700 1 -- 172.21.15.47:0/3657349955 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa15801a880 con 0x7fa168070430 2023-11-15T04:29:06.761 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.753+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1680fad90 con 0x7fa168070430 2023-11-15T04:29:06.761 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.754+0000 7fa15f7fe700 1 -- 172.21.15.47:0/3657349955 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fa15800f070 con 0x7fa168070430 2023-11-15T04:29:06.761 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.754+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa168045ad0 con 0x7fa168070430 2023-11-15T04:29:06.761 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.755+0000 7fa15f7fe700 1 --2- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa14802fd50 0x7fa148032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:06.762 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.755+0000 7fa15f7fe700 1 -- 172.21.15.47:0/3657349955 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa158041460 con 0x7fa168070430 2023-11-15T04:29:06.762 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.755+0000 7fa16659c700 1 --2- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa14802fd50 0x7fa148032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:06.762 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.755+0000 7fa16659c700 1 --2- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa14802fd50 0x7fa148032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa1500096a0 tx=0x7fa150006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:06.763 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.759+0000 7fa15f7fe700 1 -- 172.21.15.47:0/3657349955 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa158019cd0 con 0x7fa168070430 2023-11-15T04:29:06.932 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:06.931+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 --> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1680ffd10 con 0x7fa14802fd50 2023-11-15T04:29:07.119 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.118+0000 7fa15f7fe700 1 -- 172.21.15.47:0/3657349955 <== mgr.14164 v2:172.21.15.47:6800/1025769174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa1680ffd10 con 0x7fa14802fd50 2023-11-15T04:29:07.145 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-15T04:29:07.146 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa14802fd50 msgr2=0x7fa148032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:07.147 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 --2- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa14802fd50 0x7fa148032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa1500096a0 tx=0x7fa150006b40).stop 2023-11-15T04:29:07.147 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 msgr2=0x7fa1680fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:07.147 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 --2- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fc360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa15800b190 tx=0x7fa158006ee0).stop 2023-11-15T04:29:07.147 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 shutdown_connections 2023-11-15T04:29:07.147 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 --2- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa14802fd50 0x7fa148032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:07.148 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 --2- 172.21.15.47:0/3657349955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168070430 0x7fa1680fc360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:07.148 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 >> 172.21.15.47:0/3657349955 conn(0x7fa1680f63c0 msgr2=0x7fa1680f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:07.148 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 shutdown_connections 2023-11-15T04:29:07.148 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.120+0000 7fa16d0eb700 1 -- 172.21.15.47:0/3657349955 wait complete. 2023-11-15T04:29:07.417 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:06.053804+0000 mgr.smithi047.zgysuz (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T04:29:07.417 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:06.054571+0000 mon.smithi047 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi047.zgysuz(active, since 1.05544s) 2023-11-15T04:29:07.417 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:06.061315+0000 mgr.smithi047.zgysuz (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T04:29:07.474 INFO:teuthology.orchestra.run.smithi047.stderr:Creating initial admin user... 2023-11-15T04:29:07.474 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqpxeeywk:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-15T04:29:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.864+0000 7f5ea3fff700 1 Processor -- start 2023-11-15T04:29:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.864+0000 7f5ea3fff700 1 -- start start 2023-11-15T04:29:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.864+0000 7f5ea3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea4100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.864+0000 7f5ea3fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea40fc5a0 con 0x7f5ea40703b0 2023-11-15T04:29:07.869 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.864+0000 7f5ea2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea4100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:07.869 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.865+0000 7f5ea2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea4100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39650/0 (socket says 172.21.15.47:39650) 2023-11-15T04:29:07.869 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.865+0000 7f5ea2ffd700 1 -- 172.21.15.47:0/642984253 learned_addr learned my addr 172.21.15.47:0/642984253 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.865+0000 7f5ea2ffd700 1 -- 172.21.15.47:0/642984253 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea41010a0 con 0x7f5ea40703b0 2023-11-15T04:29:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.865+0000 7f5ea2ffd700 1 --2- 172.21.15.47:0/642984253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea4100bb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f5e9400b5f0 tx=0x7f5e94006ee0).ready entity=mon.0 client_cookie=e6e578bdab219e41 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.865+0000 7f5ea1ffb700 1 -- 172.21.15.47:0/642984253 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9400d600 con 0x7f5ea40703b0 2023-11-15T04:29:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.865+0000 7f5ea1ffb700 1 -- 172.21.15.47:0/642984253 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5e9400dbd0 con 0x7f5ea40703b0 2023-11-15T04:29:07.871 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.865+0000 7f5ea1ffb700 1 -- 172.21.15.47:0/642984253 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e94011c00 con 0x7f5ea40703b0 2023-11-15T04:29:07.871 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.866+0000 7f5ea3fff700 1 -- 172.21.15.47:0/642984253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 msgr2=0x7f5ea4100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:07.871 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.866+0000 7f5ea3fff700 1 --2- 172.21.15.47:0/642984253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea4100bb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f5e9400b5f0 tx=0x7f5e94006ee0).stop 2023-11-15T04:29:07.871 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.866+0000 7f5ea3fff700 1 -- 172.21.15.47:0/642984253 shutdown_connections 2023-11-15T04:29:07.871 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.866+0000 7f5ea3fff700 1 --2- 172.21.15.47:0/642984253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea4100bb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:07.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.866+0000 7f5ea3fff700 1 -- 172.21.15.47:0/642984253 >> 172.21.15.47:0/642984253 conn(0x7f5ea40f6210 msgr2=0x7f5ea40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:07.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.866+0000 7f5ea3fff700 1 -- 172.21.15.47:0/642984253 shutdown_connections 2023-11-15T04:29:07.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.866+0000 7f5ea3fff700 1 -- 172.21.15.47:0/642984253 wait complete. 2023-11-15T04:29:07.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.867+0000 7f5ea3fff700 1 Processor -- start 2023-11-15T04:29:07.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.867+0000 7f5ea3fff700 1 -- start start 2023-11-15T04:29:07.873 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.867+0000 7f5ea3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea410ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:07.873 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.867+0000 7f5ea3fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea410cf20 con 0x7f5ea40703b0 2023-11-15T04:29:07.873 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.867+0000 7f5ea2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea410ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:07.873 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.867+0000 7f5ea2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea410ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39652/0 (socket says 172.21.15.47:39652) 2023-11-15T04:29:07.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.867+0000 7f5ea2ffd700 1 -- 172.21.15.47:0/2454975643 learned_addr learned my addr 172.21.15.47:0/2454975643 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:07.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.867+0000 7f5ea2ffd700 1 -- 172.21.15.47:0/2454975643 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9400b200 con 0x7f5ea40703b0 2023-11-15T04:29:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.868+0000 7f5ea2ffd700 1 --2- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea410ca30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5e9400b190 tx=0x7f5e94006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.868+0000 7f5e9b7fe700 1 -- 172.21.15.47:0/2454975643 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9401b410 con 0x7f5ea40703b0 2023-11-15T04:29:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.868+0000 7f5e9b7fe700 1 -- 172.21.15.47:0/2454975643 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5e9401b9e0 con 0x7f5ea40703b0 2023-11-15T04:29:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.868+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea410d120 con 0x7f5ea40703b0 2023-11-15T04:29:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.868+0000 7f5e9b7fe700 1 -- 172.21.15.47:0/2454975643 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9401a880 con 0x7f5ea40703b0 2023-11-15T04:29:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.868+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea410d530 con 0x7f5ea40703b0 2023-11-15T04:29:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.869+0000 7f5e9b7fe700 1 -- 172.21.15.47:0/2454975643 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f5e9400f070 con 0x7f5ea40703b0 2023-11-15T04:29:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.869+0000 7f5e9b7fe700 1 --2- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5e8402ff00 0x7f5e840323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.869+0000 7f5e9b7fe700 1 -- 172.21.15.47:0/2454975643 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e94041460 con 0x7f5ea40703b0 2023-11-15T04:29:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.869+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea4045ad0 con 0x7f5ea40703b0 2023-11-15T04:29:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.869+0000 7f5ea27fc700 1 --2- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5e8402ff00 0x7f5e840323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.870+0000 7f5ea27fc700 1 --2- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5e8402ff00 0x7f5e840323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5e8c0096a0 tx=0x7f5e8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:07.874+0000 7f5e9b7fe700 1 -- 172.21.15.47:0/2454975643 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e94016070 con 0x7f5ea40703b0 2023-11-15T04:29:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.056+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 --> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5ea4058f20 con 0x7f5e8402ff00 2023-11-15T04:29:08.309 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.309+0000 7f5e9b7fe700 1 -- 172.21.15.47:0/2454975643 <== mgr.14164 v2:172.21.15.47:6800/1025769174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f5ea4058f20 con 0x7f5e8402ff00 2023-11-15T04:29:08.310 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$CJM0k.eCx092l9hiNKgwpud5W9CVTj3yS1BS/TvJlyGsIVjXZn/aq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700022548, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-15T04:29:08.312 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5e8402ff00 msgr2=0x7f5e840323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:08.312 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 --2- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5e8402ff00 0x7f5e840323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5e8c0096a0 tx=0x7f5e8c006b40).stop 2023-11-15T04:29:08.312 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 msgr2=0x7f5ea410ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 --2- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea410ca30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5e9400b190 tx=0x7f5e94006ee0).stop 2023-11-15T04:29:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 shutdown_connections 2023-11-15T04:29:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 --2- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5e8402ff00 0x7f5e840323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 --2- 172.21.15.47:0/2454975643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ea40703b0 0x7f5ea410ca30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 >> 172.21.15.47:0/2454975643 conn(0x7f5ea40f6210 msgr2=0x7f5ea40f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:08.314 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 shutdown_connections 2023-11-15T04:29:08.314 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:08.311+0000 7f5ea3fff700 1 -- 172.21.15.47:0/2454975643 wait complete. 2023-11-15T04:29:08.504 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:06.933186+0000 mgr.smithi047.zgysuz (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:29:08.504 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:07.117030+0000 mon.smithi047 (mon.0) 104 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:08.505 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:07.118914+0000 mon.smithi047 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:08.505 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:07.684535+0000 mon.smithi047 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi047.zgysuz(active, since 2s) 2023-11-15T04:29:08.672 INFO:teuthology.orchestra.run.smithi047.stderr:Fetching dashboard port number... 2023-11-15T04:29:08.672 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpp70flr7h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv3y72e6l:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-15T04:29:09.063 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.062+0000 7f90cf2cf700 1 Processor -- start 2023-11-15T04:29:09.064 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.062+0000 7f90cf2cf700 1 -- start start 2023-11-15T04:29:09.064 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.062+0000 7f90cf2cf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:09.064 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.062+0000 7f90cf2cf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c80fee10 con 0x7f90c80703f0 2023-11-15T04:29:09.067 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.063+0000 7f90cd06b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:09.067 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.063+0000 7f90cd06b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39668/0 (socket says 172.21.15.47:39668) 2023-11-15T04:29:09.068 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.063+0000 7f90cd06b700 1 -- 172.21.15.47:0/2283064179 learned_addr learned my addr 172.21.15.47:0/2283064179 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:09.068 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.063+0000 7f90cd06b700 1 -- 172.21.15.47:0/2283064179 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c80fef50 con 0x7f90c80703f0 2023-11-15T04:29:09.068 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.063+0000 7f90cd06b700 1 --2- 172.21.15.47:0/2283064179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fe920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f90c400b5f0 tx=0x7f90c4006ee0).ready entity=mon.0 client_cookie=79c2c9d35f9756c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:09.068 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.063+0000 7f90b7fff700 1 -- 172.21.15.47:0/2283064179 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c400d600 con 0x7f90c80703f0 2023-11-15T04:29:09.068 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.063+0000 7f90b7fff700 1 -- 172.21.15.47:0/2283064179 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f90c400dbd0 con 0x7f90c80703f0 2023-11-15T04:29:09.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.063+0000 7f90b7fff700 1 -- 172.21.15.47:0/2283064179 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c4011c00 con 0x7f90c80703f0 2023-11-15T04:29:09.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.064+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2283064179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 msgr2=0x7f90c80fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:09.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.064+0000 7f90cf2cf700 1 --2- 172.21.15.47:0/2283064179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fe920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f90c400b5f0 tx=0x7f90c4006ee0).stop 2023-11-15T04:29:09.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.064+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2283064179 shutdown_connections 2023-11-15T04:29:09.069 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.064+0000 7f90cf2cf700 1 --2- 172.21.15.47:0/2283064179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fe920 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:09.070 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.064+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2283064179 >> 172.21.15.47:0/2283064179 conn(0x7f90c80f63c0 msgr2=0x7f90c80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:09.070 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.064+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2283064179 shutdown_connections 2023-11-15T04:29:09.070 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.064+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2283064179 wait complete. 2023-11-15T04:29:09.070 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.065+0000 7f90cf2cf700 1 Processor -- start 2023-11-15T04:29:09.070 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.065+0000 7f90cf2cf700 1 -- start start 2023-11-15T04:29:09.071 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.065+0000 7f90cf2cf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:09.071 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.065+0000 7f90cf2cf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c80fa820 con 0x7f90c80703f0 2023-11-15T04:29:09.071 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.065+0000 7f90cd06b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:09.071 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.065+0000 7f90cd06b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39676/0 (socket says 172.21.15.47:39676) 2023-11-15T04:29:09.071 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.065+0000 7f90cd06b700 1 -- 172.21.15.47:0/2788331523 learned_addr learned my addr 172.21.15.47:0/2788331523 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:09.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.066+0000 7f90cd06b700 1 -- 172.21.15.47:0/2788331523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c400b200 con 0x7f90c80703f0 2023-11-15T04:29:09.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.066+0000 7f90cd06b700 1 --2- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fc3a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f90c400e040 tx=0x7f90c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:09.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.066+0000 7f90b5ffb700 1 -- 172.21.15.47:0/2788331523 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c400d9f0 con 0x7f90c80703f0 2023-11-15T04:29:09.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.066+0000 7f90b5ffb700 1 -- 172.21.15.47:0/2788331523 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f90c401a9f0 con 0x7f90c80703f0 2023-11-15T04:29:09.072 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.066+0000 7f90b5ffb700 1 -- 172.21.15.47:0/2788331523 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c401b8c0 con 0x7f90c80703f0 2023-11-15T04:29:09.073 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.066+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c80fa9c0 con 0x7f90c80703f0 2023-11-15T04:29:09.073 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.066+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c80fadd0 con 0x7f90c80703f0 2023-11-15T04:29:09.074 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.067+0000 7f90b5ffb700 1 -- 172.21.15.47:0/2788331523 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f90c400f070 con 0x7f90c80703f0 2023-11-15T04:29:09.074 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.067+0000 7f90b5ffb700 1 --2- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f90b002ff00 0x7f90b00323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:09.074 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.067+0000 7f90b5ffb700 1 -- 172.21.15.47:0/2788331523 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f90c4041530 con 0x7f90c80703f0 2023-11-15T04:29:09.074 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.067+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c8045ad0 con 0x7f90c80703f0 2023-11-15T04:29:09.075 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.067+0000 7f90cc86a700 1 --2- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f90b002ff00 0x7f90b00323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:09.075 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.068+0000 7f90cc86a700 1 --2- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f90b002ff00 0x7f90b00323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f90b80096a0 tx=0x7f90b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:09.075 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.072+0000 7f90b5ffb700 1 -- 172.21.15.47:0/2788331523 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90c4016070 con 0x7f90c80703f0 2023-11-15T04:29:09.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.237+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f90c80ffbd0 con 0x7f90c80703f0 2023-11-15T04:29:09.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.238+0000 7f90b5ffb700 1 -- 172.21.15.47:0/2788331523 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f90c4025460 con 0x7f90c80703f0 2023-11-15T04:29:09.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 8443 2023-11-15T04:29:09.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f90b002ff00 msgr2=0x7f90b00323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:09.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 --2- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f90b002ff00 0x7f90b00323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f90b80096a0 tx=0x7f90b8006b40).stop 2023-11-15T04:29:09.240 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 msgr2=0x7f90c80fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:09.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 --2- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fc3a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f90c400e040 tx=0x7f90c4006ee0).stop 2023-11-15T04:29:09.241 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 shutdown_connections 2023-11-15T04:29:09.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 --2- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f90b002ff00 0x7f90b00323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:09.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 --2- 172.21.15.47:0/2788331523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90c80703f0 0x7f90c80fc3a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:09.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 >> 172.21.15.47:0/2788331523 conn(0x7f90c80f63c0 msgr2=0x7f90c80f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:09.242 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 shutdown_connections 2023-11-15T04:29:09.243 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2023-11-15T04:29:09.239+0000 7f90cf2cf700 1 -- 172.21.15.47:0/2788331523 wait complete. 2023-11-15T04:29:09.533 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:08.058081+0000 mgr.smithi047.zgysuz (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:29:09.533 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:08.309709+0000 mon.smithi047 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:09.534 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:09.238830+0000 mon.smithi047 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.47:0/2788331523' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-15T04:29:09.622 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:29:09.633 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2023-11-15T04:29:09.634 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:29:09.643 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2023-11-15T04:29:09.643 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2023-11-15T04:29:09.644 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-15T04:29:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:Ceph Dashboard is now available at: 2023-11-15T04:29:09.648 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:29:09.648 INFO:teuthology.orchestra.run.smithi047.stderr: URL: https://smithi047.front.sepia.ceph.com:8443/ 2023-11-15T04:29:09.648 INFO:teuthology.orchestra.run.smithi047.stderr: User: admin 2023-11-15T04:29:09.648 INFO:teuthology.orchestra.run.smithi047.stderr: Password: 7o02zptfqc 2023-11-15T04:29:09.648 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:29:09.649 INFO:teuthology.orchestra.run.smithi047.stderr:You can access the Ceph CLI with: 2023-11-15T04:29:09.649 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:29:09.649 INFO:teuthology.orchestra.run.smithi047.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-15T04:29:09.649 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:29:09.649 INFO:teuthology.orchestra.run.smithi047.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-15T04:29:09.650 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:29:09.650 INFO:teuthology.orchestra.run.smithi047.stderr: ceph telemetry on 2023-11-15T04:29:09.650 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:29:09.650 INFO:teuthology.orchestra.run.smithi047.stderr:For more information see: 2023-11-15T04:29:09.651 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:29:09.651 INFO:teuthology.orchestra.run.smithi047.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-15T04:29:09.651 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:29:09.651 INFO:teuthology.orchestra.run.smithi047.stderr:Bootstrap complete. 2023-11-15T04:29:09.651 INFO:teuthology.orchestra.run.smithi047.stderr:Releasing lock 140321730787816 on /run/cephadm/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:29:09.652 INFO:teuthology.orchestra.run.smithi047.stderr:Lock 140321730787816 released on /run/cephadm/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:29:09.690 INFO:tasks.cephadm:Fetching config... 2023-11-15T04:29:09.690 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:29:09.691 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-15T04:29:09.752 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-15T04:29:09.753 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:29:09.753 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-15T04:29:09.814 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-15T04:29:09.815 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:29:09.815 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/mon.smithi047/keyring of=/dev/stdout 2023-11-15T04:29:09.889 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-15T04:29:09.890 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2023-11-15T04:29:09.890 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-15T04:29:09.952 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-15T04:29:09.953 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCy0qrdUxrxnhvk+iYQwiUpKs7qnJhVAaM3q2FtgBHCoDfx8b1Os+kPjXiVCPoYC7QcpLXaiviRBNsUK7k86PYiJuHUy8Tf2oL0+A04yt0Y4SBjthSTHquqx5Ld9/bKqU6JXRHPTtSiEgyvOzkEgek37x+BawoJ3iwuGGhBigneviwea9Zs9KpexCCrCBfqsoYBdspsmIACPt7pKF5GXeaAEdtXYIRgmfqWG0F5BR++OoYoojR/gQ2eiAf2MshaIkYBvvHjeqQJ3wuOSbZQD31LqzCPGXgrZ6W4VlKNpEkFnG0sXi+RbAn4NunFamrLLZeKCGvsN2sQleJDRKbVFWwexvsf5O0kzJiO+06MAZtgkwcESJMOlRRQSuU7rI382dl5U6SzZj9o/ye/Bo27/VKSqDTu7o3GVl3p0ABeZqDG8baEBh+d8jauYGSvVzFQSbp9Q+AMsBiski+SvzvppZ8+9yV+b+v4WJ2Y88NZGQceWkGnMrY6jqe0xOXDQ/XSABE= ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T04:29:10.038 INFO:teuthology.orchestra.run.smithi047.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCy0qrdUxrxnhvk+iYQwiUpKs7qnJhVAaM3q2FtgBHCoDfx8b1Os+kPjXiVCPoYC7QcpLXaiviRBNsUK7k86PYiJuHUy8Tf2oL0+A04yt0Y4SBjthSTHquqx5Ld9/bKqU6JXRHPTtSiEgyvOzkEgek37x+BawoJ3iwuGGhBigneviwea9Zs9KpexCCrCBfqsoYBdspsmIACPt7pKF5GXeaAEdtXYIRgmfqWG0F5BR++OoYoojR/gQ2eiAf2MshaIkYBvvHjeqQJ3wuOSbZQD31LqzCPGXgrZ6W4VlKNpEkFnG0sXi+RbAn4NunFamrLLZeKCGvsN2sQleJDRKbVFWwexvsf5O0kzJiO+06MAZtgkwcESJMOlRRQSuU7rI382dl5U6SzZj9o/ye/Bo27/VKSqDTu7o3GVl3p0ABeZqDG8baEBh+d8jauYGSvVzFQSbp9Q+AMsBiski+SvzvppZ8+9yV+b+v4WJ2Y88NZGQceWkGnMrY6jqe0xOXDQ/XSABE= ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:29:10.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCy0qrdUxrxnhvk+iYQwiUpKs7qnJhVAaM3q2FtgBHCoDfx8b1Os+kPjXiVCPoYC7QcpLXaiviRBNsUK7k86PYiJuHUy8Tf2oL0+A04yt0Y4SBjthSTHquqx5Ld9/bKqU6JXRHPTtSiEgyvOzkEgek37x+BawoJ3iwuGGhBigneviwea9Zs9KpexCCrCBfqsoYBdspsmIACPt7pKF5GXeaAEdtXYIRgmfqWG0F5BR++OoYoojR/gQ2eiAf2MshaIkYBvvHjeqQJ3wuOSbZQD31LqzCPGXgrZ6W4VlKNpEkFnG0sXi+RbAn4NunFamrLLZeKCGvsN2sQleJDRKbVFWwexvsf5O0kzJiO+06MAZtgkwcESJMOlRRQSuU7rI382dl5U6SzZj9o/ye/Bo27/VKSqDTu7o3GVl3p0ABeZqDG8baEBh+d8jauYGSvVzFQSbp9Q+AMsBiski+SvzvppZ8+9yV+b+v4WJ2Y88NZGQceWkGnMrY6jqe0xOXDQ/XSABE= ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T04:29:10.095 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCy0qrdUxrxnhvk+iYQwiUpKs7qnJhVAaM3q2FtgBHCoDfx8b1Os+kPjXiVCPoYC7QcpLXaiviRBNsUK7k86PYiJuHUy8Tf2oL0+A04yt0Y4SBjthSTHquqx5Ld9/bKqU6JXRHPTtSiEgyvOzkEgek37x+BawoJ3iwuGGhBigneviwea9Zs9KpexCCrCBfqsoYBdspsmIACPt7pKF5GXeaAEdtXYIRgmfqWG0F5BR++OoYoojR/gQ2eiAf2MshaIkYBvvHjeqQJ3wuOSbZQD31LqzCPGXgrZ6W4VlKNpEkFnG0sXi+RbAn4NunFamrLLZeKCGvsN2sQleJDRKbVFWwexvsf5O0kzJiO+06MAZtgkwcESJMOlRRQSuU7rI382dl5U6SzZj9o/ye/Bo27/VKSqDTu7o3GVl3p0ABeZqDG8baEBh+d8jauYGSvVzFQSbp9Q+AMsBiski+SvzvppZ8+9yV+b+v4WJ2Y88NZGQceWkGnMrY6jqe0xOXDQ/XSABE= ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 2023-11-15T04:29:10.107 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-15T04:29:11.294 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 -- 172.21.15.47:0/368461554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 msgr2=0x7fa7c00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:11.294 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 --2- 172.21.15.47:0/368461554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 0x7fa7c00f7d00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa7b0009fb0 tx=0x7fa7b0009b30).stop 2023-11-15T04:29:11.295 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 -- 172.21.15.47:0/368461554 shutdown_connections 2023-11-15T04:29:11.295 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 --2- 172.21.15.47:0/368461554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 0x7fa7c00f7d00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:11.295 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 -- 172.21.15.47:0/368461554 >> 172.21.15.47:0/368461554 conn(0x7fa7c00f0120 msgr2=0x7fa7c00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:11.295 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 -- 172.21.15.47:0/368461554 shutdown_connections 2023-11-15T04:29:11.295 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 -- 172.21.15.47:0/368461554 wait complete. 2023-11-15T04:29:11.296 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 Processor -- start 2023-11-15T04:29:11.296 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 -- start start 2023-11-15T04:29:11.296 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 0x7fa7c0106490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:11.296 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0106980 con 0x7fa7c00f78e0 2023-11-15T04:29:11.297 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7bffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 0x7fa7c0106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:11.297 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7bffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 0x7fa7c0106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39680/0 (socket says 172.21.15.47:39680) 2023-11-15T04:29:11.297 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7bffff700 1 -- 172.21.15.47:0/3539435582 learned_addr learned my addr 172.21.15.47:0/3539435582 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:11.297 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7bffff700 1 -- 172.21.15.47:0/3539435582 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0005040 con 0x7fa7c00f78e0 2023-11-15T04:29:11.298 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7bffff700 1 --2- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 0x7fa7c0106490 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa7b0002040 tx=0x7fa7b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:11.298 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7bcff9700 1 -- 172.21.15.47:0/3539435582 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7b00095f0 con 0x7fa7c00f78e0 2023-11-15T04:29:11.298 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c0106b80 con 0x7fa7c00f78e0 2023-11-15T04:29:11.298 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7bcff9700 1 -- 172.21.15.47:0/3539435582 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa7b0007e30 con 0x7fa7c00f78e0 2023-11-15T04:29:11.298 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7bcff9700 1 -- 172.21.15.47:0/3539435582 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7b0014d20 con 0x7fa7c00f78e0 2023-11-15T04:29:11.299 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.293+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c0107010 con 0x7fa7c00f78e0 2023-11-15T04:29:11.300 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.294+0000 7fa7bcff9700 1 -- 172.21.15.47:0/3539435582 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7b0025070 con 0x7fa7c00f78e0 2023-11-15T04:29:11.300 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.294+0000 7fa7bcff9700 1 --2- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7a802ff60 0x7fa7a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:11.300 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.294+0000 7fa7bcff9700 1 -- 172.21.15.47:0/3539435582 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7b0040920 con 0x7fa7c00f78e0 2023-11-15T04:29:11.300 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.295+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c0045ad0 con 0x7fa7c00f78e0 2023-11-15T04:29:11.301 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.295+0000 7fa7bf7fe700 1 --2- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7a802ff60 0x7fa7a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:11.301 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.295+0000 7fa7bf7fe700 1 --2- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7a802ff60 0x7fa7a8032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa7b40096a0 tx=0x7fa7b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:11.301 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.300+0000 7fa7bcff9700 1 -- 172.21.15.47:0/3539435582 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7b000f4e0 con 0x7fa7c00f78e0 2023-11-15T04:29:11.466 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.465+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa7c0107da0 con 0x7fa7c00f78e0 2023-11-15T04:29:11.469 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.468+0000 7fa7bcff9700 1 -- 172.21.15.47:0/3539435582 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa7b000f300 con 0x7fa7c00f78e0 2023-11-15T04:29:11.471 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7a802ff60 msgr2=0x7fa7a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:11.471 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 --2- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7a802ff60 0x7fa7a8032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa7b40096a0 tx=0x7fa7b4006b40).stop 2023-11-15T04:29:11.471 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 msgr2=0x7fa7c0106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:11.471 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 --2- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 0x7fa7c0106490 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa7b0002040 tx=0x7fa7b000b040).stop 2023-11-15T04:29:11.471 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 shutdown_connections 2023-11-15T04:29:11.472 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 --2- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7a802ff60 0x7fa7a8032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:11.472 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 --2- 172.21.15.47:0/3539435582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7c00f78e0 0x7fa7c0106490 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:11.472 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 >> 172.21.15.47:0/3539435582 conn(0x7fa7c00f0120 msgr2=0x7fa7c00f0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:11.472 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 shutdown_connections 2023-11-15T04:29:11.472 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:11.470+0000 7fa7c5286700 1 -- 172.21.15.47:0/3539435582 wait complete. 2023-11-15T04:29:11.850 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2023-11-15T04:29:11.850 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-15T04:29:11.851 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2023-11-15T04:29:11.872 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-15T04:29:11.873 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-15T04:29:11.933 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2023-11-15T04:29:11.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi190 2023-11-15T04:29:12.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:12 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:11.079074+0000 mon.smithi047 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi047.zgysuz(active, since 6s) 2023-11-15T04:29:12.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:12 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:11.469539+0000 mon.smithi047 (mon.0) 110 : audit [INF] from='client.? 172.21.15.47:0/3539435582' entity='client.admin' 2023-11-15T04:29:14.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:13.151584+0000 mon.smithi047 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:14.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:13.397247+0000 mon.smithi047 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:14.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:14.147621+0000 mon.smithi047 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:17.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:16.435820+0000 mon.smithi047 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:17.757 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:16.734487+0000 mon.smithi047 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:17.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:16.736691+0000 mon.smithi047 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:17.758 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:29:16.743284+0000 mgr.smithi047.zgysuz (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi047 on smithi047 2023-11-15T04:29:26.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:25.055870+0000 mgr.smithi047.zgysuz (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:27.056240+0000 mgr.smithi047.zgysuz (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:30.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:29.056515+0000 mgr.smithi047.zgysuz (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:31.056740+0000 mgr.smithi047.zgysuz (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:33.185 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.181+0000 7f0d3304c700 1 -- 172.21.15.190:0/895022452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 msgr2=0x7f0d2c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:33.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.181+0000 7f0d3304c700 1 --2- 172.21.15.190:0/895022452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f8f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0d28009fb0 tx=0x7f0d28009b30).stop 2023-11-15T04:29:33.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.181+0000 7f0d3304c700 1 -- 172.21.15.190:0/895022452 shutdown_connections 2023-11-15T04:29:33.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.181+0000 7f0d3304c700 1 --2- 172.21.15.190:0/895022452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f8f20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:33.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.181+0000 7f0d3304c700 1 -- 172.21.15.190:0/895022452 >> 172.21.15.190:0/895022452 conn(0x7f0d2c0f09a0 msgr2=0x7f0d2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:33.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.181+0000 7f0d3304c700 1 -- 172.21.15.190:0/895022452 shutdown_connections 2023-11-15T04:29:33.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.181+0000 7f0d3304c700 1 -- 172.21.15.190:0/895022452 wait complete. 2023-11-15T04:29:33.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.182+0000 7f0d3304c700 1 Processor -- start 2023-11-15T04:29:33.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.182+0000 7f0d3304c700 1 -- start start 2023-11-15T04:29:33.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.182+0000 7f0d3304c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:33.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.182+0000 7f0d3304c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2c0f4e20 con 0x7f0d2c0601e0 2023-11-15T04:29:33.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.182+0000 7f0d3204a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:33.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.182+0000 7f0d3204a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:59342/0 (socket says 172.21.15.190:59342) 2023-11-15T04:29:33.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.182+0000 7f0d3204a700 1 -- 172.21.15.190:0/232725715 learned_addr learned my addr 172.21.15.190:0/232725715 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:29:33.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.182+0000 7f0d3204a700 1 -- 172.21.15.190:0/232725715 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d28005040 con 0x7f0d2c0601e0 2023-11-15T04:29:33.191 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.182+0000 7f0d3204a700 1 --2- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f69a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0d28013510 tx=0x7f0d2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:33.191 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.183+0000 7f0d22ffd700 1 -- 172.21.15.190:0/232725715 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d28009320 con 0x7f0d2c0601e0 2023-11-15T04:29:33.191 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.183+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2c0f4fc0 con 0x7f0d2c0601e0 2023-11-15T04:29:33.191 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.183+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2c0f5450 con 0x7f0d2c0601e0 2023-11-15T04:29:33.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.183+0000 7f0d22ffd700 1 -- 172.21.15.190:0/232725715 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d28013f00 con 0x7f0d2c0601e0 2023-11-15T04:29:33.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.183+0000 7f0d22ffd700 1 -- 172.21.15.190:0/232725715 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d28014d60 con 0x7f0d2c0601e0 2023-11-15T04:29:33.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.183+0000 7f0d22ffd700 1 -- 172.21.15.190:0/232725715 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d280095f0 con 0x7f0d2c0601e0 2023-11-15T04:29:33.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.183+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2c059100 con 0x7f0d2c0601e0 2023-11-15T04:29:33.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.184+0000 7f0d22ffd700 1 --2- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0d1802ff60 0x7f0d18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:33.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.184+0000 7f0d22ffd700 1 -- 172.21.15.190:0/232725715 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d280410b0 con 0x7f0d2c0601e0 2023-11-15T04:29:33.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.184+0000 7f0d31849700 1 --2- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0d1802ff60 0x7f0d18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:33.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.184+0000 7f0d31849700 1 --2- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0d1802ff60 0x7f0d18032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0d1c0096a0 tx=0x7f0d1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:33.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.188+0000 7f0d22ffd700 1 -- 172.21.15.190:0/232725715 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d28014420 con 0x7f0d2c0601e0 2023-11-15T04:29:33.362 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.361+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 --> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d2c0f5ca0 con 0x7f0d1802ff60 2023-11-15T04:29:33.969 INFO:teuthology.orchestra.run.smithi190.stdout:Added host 'smithi190' 2023-11-15T04:29:33.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.967+0000 7f0d22ffd700 1 -- 172.21.15.190:0/232725715 <== mgr.14164 v2:172.21.15.47:6800/1025769174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f0d2c0f5ca0 con 0x7f0d1802ff60 2023-11-15T04:29:33.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.969+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0d1802ff60 msgr2=0x7f0d18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:33.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.969+0000 7f0d3304c700 1 --2- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0d1802ff60 0x7f0d18032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0d1c0096a0 tx=0x7f0d1c006b40).stop 2023-11-15T04:29:33.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.969+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 msgr2=0x7f0d2c0f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:33.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.969+0000 7f0d3304c700 1 --2- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f69a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0d28013510 tx=0x7f0d2800b040).stop 2023-11-15T04:29:33.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.969+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 shutdown_connections 2023-11-15T04:29:33.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.969+0000 7f0d3304c700 1 --2- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0d1802ff60 0x7f0d18032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:33.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.970+0000 7f0d3304c700 1 --2- 172.21.15.190:0/232725715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f69a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:33.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.970+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 >> 172.21.15.190:0/232725715 conn(0x7f0d2c0f09a0 msgr2=0x7f0d2c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:33.975 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.970+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 shutdown_connections 2023-11-15T04:29:33.975 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:33.970+0000 7f0d3304c700 1 -- 172.21.15.190:0/232725715 wait complete. 2023-11-15T04:29:34.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-15T04:29:34.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:33.057073+0000 mgr.smithi047.zgysuz (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:34.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:33.967774+0000 mon.smithi047 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:35.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.415+0000 7f21206b4700 1 -- 172.21.15.190:0/1539031406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 msgr2=0x7f21180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:35.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.415+0000 7f21206b4700 1 --2- 172.21.15.190:0/1539031406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 0x7f21180f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f2108009fb0 tx=0x7f2108009b30).stop 2023-11-15T04:29:35.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.415+0000 7f21206b4700 1 -- 172.21.15.190:0/1539031406 shutdown_connections 2023-11-15T04:29:35.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.415+0000 7f21206b4700 1 --2- 172.21.15.190:0/1539031406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 0x7f21180f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:35.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.415+0000 7f21206b4700 1 -- 172.21.15.190:0/1539031406 >> 172.21.15.190:0/1539031406 conn(0x7f21180f1c70 msgr2=0x7f21180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:35.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.415+0000 7f21206b4700 1 -- 172.21.15.190:0/1539031406 shutdown_connections 2023-11-15T04:29:35.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.415+0000 7f21206b4700 1 -- 172.21.15.190:0/1539031406 wait complete. 2023-11-15T04:29:35.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.416+0000 7f21206b4700 1 Processor -- start 2023-11-15T04:29:35.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.416+0000 7f21206b4700 1 -- start start 2023-11-15T04:29:35.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.416+0000 7f21206b4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 0x7f2118106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:35.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.416+0000 7f21206b4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2118106c20 con 0x7f21180f6510 2023-11-15T04:29:35.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.416+0000 7f211e450700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 0x7f2118106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:35.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.416+0000 7f211e450700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 0x7f2118106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:59346/0 (socket says 172.21.15.190:59346) 2023-11-15T04:29:35.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.416+0000 7f211e450700 1 -- 172.21.15.190:0/4005643403 learned_addr learned my addr 172.21.15.190:0/4005643403 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:29:35.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.417+0000 7f211e450700 1 -- 172.21.15.190:0/4005643403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2108005040 con 0x7f21180f6510 2023-11-15T04:29:35.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.417+0000 7f211e450700 1 --2- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 0x7f2118106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f2108013510 tx=0x7f210800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:35.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.417+0000 7f210effd700 1 -- 172.21.15.190:0/4005643403 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2108009320 con 0x7f21180f6510 2023-11-15T04:29:35.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.417+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2118106e20 con 0x7f21180f6510 2023-11-15T04:29:35.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.417+0000 7f210effd700 1 -- 172.21.15.190:0/4005643403 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2108013f00 con 0x7f21180f6510 2023-11-15T04:29:35.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.417+0000 7f210effd700 1 -- 172.21.15.190:0/4005643403 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2108014d90 con 0x7f21180f6510 2023-11-15T04:29:35.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.417+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21181072b0 con 0x7f21180f6510 2023-11-15T04:29:35.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.418+0000 7f210effd700 1 -- 172.21.15.190:0/4005643403 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2108014420 con 0x7f21180f6510 2023-11-15T04:29:35.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.418+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2118045ad0 con 0x7f21180f6510 2023-11-15T04:29:35.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.418+0000 7f210effd700 1 --2- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f210402ff60 0x7f2104032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:35.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.418+0000 7f210effd700 1 -- 172.21.15.190:0/4005643403 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21080410c0 con 0x7f21180f6510 2023-11-15T04:29:35.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.418+0000 7f211dc4f700 1 --2- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f210402ff60 0x7f2104032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:35.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.419+0000 7f211dc4f700 1 --2- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f210402ff60 0x7f2104032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f21140096a0 tx=0x7f2114006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:35.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.423+0000 7f210effd700 1 -- 172.21.15.190:0/4005643403 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21080095f0 con 0x7f21180f6510 2023-11-15T04:29:35.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:35 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:33.364200+0000 mgr.smithi047.zgysuz (mgr.14164) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:29:35.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:35 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:29:33.968205+0000 mgr.smithi047.zgysuz (mgr.14164) 12 : cephadm [INF] Added host smithi190 2023-11-15T04:29:35.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.592+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 --> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21180f2e70 con 0x7f210402ff60 2023-11-15T04:29:35.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.594+0000 7f210effd700 1 -- 172.21.15.190:0/4005643403 <== mgr.14164 v2:172.21.15.47:6800/1025769174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f21180f2e70 con 0x7f210402ff60 2023-11-15T04:29:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:29:35.595 INFO:teuthology.orchestra.run.smithi190.stdout:[{"addr": "172.21.15.47", "hostname": "smithi047", "labels": [], "status": ""}, {"addr": "smithi190", "hostname": "smithi190", "labels": [], "status": ""}] 2023-11-15T04:29:35.596 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f210402ff60 msgr2=0x7f2104032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:35.597 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 --2- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f210402ff60 0x7f2104032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f21140096a0 tx=0x7f2114006b40).stop 2023-11-15T04:29:35.597 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 msgr2=0x7f2118106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:35.597 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 --2- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 0x7f2118106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f2108013510 tx=0x7f210800b040).stop 2023-11-15T04:29:35.598 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 shutdown_connections 2023-11-15T04:29:35.598 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 --2- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f210402ff60 0x7f2104032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:35.598 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 --2- 172.21.15.190:0/4005643403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21180f6510 0x7f2118106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:35.598 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 >> 172.21.15.190:0/4005643403 conn(0x7f21180f1c70 msgr2=0x7f21180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:35.599 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 shutdown_connections 2023-11-15T04:29:35.599 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:35.595+0000 7f21206b4700 1 -- 172.21.15.190:0/4005643403 wait complete. 2023-11-15T04:29:35.969 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-15T04:29:35.969 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-15T04:29:36.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:35.057320+0000 mgr.smithi047.zgysuz (mgr.14164) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.165+0000 7f017974d700 1 -- 172.21.15.47:0/874406242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 msgr2=0x7f01740fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.165+0000 7f017974d700 1 --2- 172.21.15.47:0/874406242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 0x7f01740fb3a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f015c009fb0 tx=0x7f015c009b30).stop 2023-11-15T04:29:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.165+0000 7f017974d700 1 -- 172.21.15.47:0/874406242 shutdown_connections 2023-11-15T04:29:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.165+0000 7f017974d700 1 --2- 172.21.15.47:0/874406242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 0x7f01740fb3a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.165+0000 7f017974d700 1 -- 172.21.15.47:0/874406242 >> 172.21.15.47:0/874406242 conn(0x7f01740f09a0 msgr2=0x7f01740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.165+0000 7f017974d700 1 -- 172.21.15.47:0/874406242 shutdown_connections 2023-11-15T04:29:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.165+0000 7f017974d700 1 -- 172.21.15.47:0/874406242 wait complete. 2023-11-15T04:29:37.169 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.166+0000 7f017974d700 1 Processor -- start 2023-11-15T04:29:37.169 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.166+0000 7f017974d700 1 -- start start 2023-11-15T04:29:37.169 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.166+0000 7f017974d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 0x7f01741086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:37.169 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.166+0000 7f017974d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0174108be0 con 0x7f0174060160 2023-11-15T04:29:37.170 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f0173fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 0x7f01741086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:37.170 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f0173fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 0x7f01741086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49350/0 (socket says 172.21.15.47:49350) 2023-11-15T04:29:37.170 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f0173fff700 1 -- 172.21.15.47:0/1464854995 learned_addr learned my addr 172.21.15.47:0/1464854995 (peer_addr_for_me v2:172.21.15.47:0/0) 2023-11-15T04:29:37.170 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f0173fff700 1 -- 172.21.15.47:0/1464854995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c005040 con 0x7f0174060160 2023-11-15T04:29:37.171 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f0173fff700 1 --2- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 0x7f01741086f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f015c013510 tx=0x7f015c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:37.171 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f0170ff9700 1 -- 172.21.15.47:0/1464854995 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f015c009320 con 0x7f0174060160 2023-11-15T04:29:37.171 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f0170ff9700 1 -- 172.21.15.47:0/1464854995 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f015c013f00 con 0x7f0174060160 2023-11-15T04:29:37.171 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0174108de0 con 0x7f0174060160 2023-11-15T04:29:37.171 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f0170ff9700 1 -- 172.21.15.47:0/1464854995 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f015c014d90 con 0x7f0174060160 2023-11-15T04:29:37.172 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.167+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0174109270 con 0x7f0174060160 2023-11-15T04:29:37.172 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.168+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0174045ad0 con 0x7f0174060160 2023-11-15T04:29:37.173 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.169+0000 7f0170ff9700 1 -- 172.21.15.47:0/1464854995 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f015c0095f0 con 0x7f0174060160 2023-11-15T04:29:37.173 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.169+0000 7f0170ff9700 1 --2- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f016002ff60 0x7f0160032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:37.173 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.169+0000 7f0170ff9700 1 -- 172.21.15.47:0/1464854995 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f015c041180 con 0x7f0174060160 2023-11-15T04:29:37.173 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.169+0000 7f01737fe700 1 --2- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f016002ff60 0x7f0160032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:37.173 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.170+0000 7f01737fe700 1 --2- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f016002ff60 0x7f0160032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f01640096a0 tx=0x7f0164006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:37.174 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.173+0000 7f0170ff9700 1 -- 172.21.15.47:0/1464854995 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f015c014420 con 0x7f0174060160 2023-11-15T04:29:37.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:35.594822+0000 mgr.smithi047.zgysuz (mgr.14164) 14 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-15T04:29:37.344 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:37.343+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f017410a120 con 0x7f0174060160 2023-11-15T04:29:38.116 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.115+0000 7f0170ff9700 1 -- 172.21.15.47:0/1464854995 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f015c02b070 con 0x7f0174060160 2023-11-15T04:29:38.119 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f016002ff60 msgr2=0x7f0160032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:38.119 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 --2- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f016002ff60 0x7f0160032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f01640096a0 tx=0x7f0164006b40).stop 2023-11-15T04:29:38.119 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 msgr2=0x7f01741086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:38.119 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 --2- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 0x7f01741086f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f015c013510 tx=0x7f015c00b040).stop 2023-11-15T04:29:38.119 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 shutdown_connections 2023-11-15T04:29:38.120 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 --2- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f016002ff60 0x7f0160032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:38.120 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 --2- 172.21.15.47:0/1464854995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0174060160 0x7f01741086f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:38.120 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 >> 172.21.15.47:0/1464854995 conn(0x7f01740f09a0 msgr2=0x7f01740f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:38.120 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 shutdown_connections 2023-11-15T04:29:38.120 INFO:teuthology.orchestra.run.smithi047.stderr:2023-11-15T04:29:38.118+0000 7f017974d700 1 -- 172.21.15.47:0/1464854995 wait complete. 2023-11-15T04:29:38.121 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2023-11-15T04:29:38.404 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:38 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:37.057631+0000 mgr.smithi047.zgysuz (mgr.14164) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:38.405 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:38 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:37.345178+0000 mon.smithi047 (mon.0) 118 : audit [INF] from='client.? 172.21.15.47:0/1464854995' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-15T04:29:38.504 INFO:tasks.cephadm:Adding mon.smithi047 on smithi047 2023-11-15T04:29:38.504 INFO:tasks.cephadm:Adding mon.smithi190 on smithi190 2023-11-15T04:29:38.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi047:172.21.15.47=smithi047;smithi190:172.21.15.190=smithi190' 2023-11-15T04:29:39.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:39 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:38.116656+0000 mon.smithi047 (mon.0) 119 : audit [INF] from='client.? 172.21.15.47:0/1464854995' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-15T04:29:39.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:39 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:38.116712+0000 mon.smithi047 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-15T04:29:39.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.613+0000 7f7e2759e700 1 -- 172.21.15.190:0/633342979 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 msgr2=0x7f7e280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:39.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.613+0000 7f7e2759e700 1 --2- 172.21.15.190:0/633342979 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 0x7f7e280f55c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f7e18009fb0 tx=0x7f7e18009b30).stop 2023-11-15T04:29:39.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.614+0000 7f7e2759e700 1 -- 172.21.15.190:0/633342979 shutdown_connections 2023-11-15T04:29:39.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.614+0000 7f7e2759e700 1 --2- 172.21.15.190:0/633342979 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 0x7f7e280f55c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:39.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.614+0000 7f7e2759e700 1 -- 172.21.15.190:0/633342979 >> 172.21.15.190:0/633342979 conn(0x7f7e280f09a0 msgr2=0x7f7e280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:39.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.614+0000 7f7e2759e700 1 -- 172.21.15.190:0/633342979 shutdown_connections 2023-11-15T04:29:39.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.614+0000 7f7e2759e700 1 -- 172.21.15.190:0/633342979 wait complete. 2023-11-15T04:29:39.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.614+0000 7f7e2759e700 1 Processor -- start 2023-11-15T04:29:39.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.614+0000 7f7e2759e700 1 -- start start 2023-11-15T04:29:39.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.614+0000 7f7e2759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 0x7f7e280fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:39.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.614+0000 7f7e2759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280fb5a0 con 0x7f7e280f51a0 2023-11-15T04:29:39.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e2659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 0x7f7e280fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:39.619 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e2659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 0x7f7e280fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:59370/0 (socket says 172.21.15.190:59370) 2023-11-15T04:29:39.619 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e2659c700 1 -- 172.21.15.190:0/4094841727 learned_addr learned my addr 172.21.15.190:0/4094841727 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:29:39.619 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e2659c700 1 -- 172.21.15.190:0/4094841727 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e18005040 con 0x7f7e280f51a0 2023-11-15T04:29:39.619 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e2659c700 1 --2- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 0x7f7e280fd120 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f7e18013510 tx=0x7f7e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:39.619 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e16ffd700 1 -- 172.21.15.190:0/4094841727 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e18009320 con 0x7f7e280f51a0 2023-11-15T04:29:39.620 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e280fb740 con 0x7f7e280f51a0 2023-11-15T04:29:39.620 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e16ffd700 1 -- 172.21.15.190:0/4094841727 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e18013f00 con 0x7f7e280f51a0 2023-11-15T04:29:39.620 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e16ffd700 1 -- 172.21.15.190:0/4094841727 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e18014dc0 con 0x7f7e280f51a0 2023-11-15T04:29:39.620 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.615+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e280fbbd0 con 0x7f7e280f51a0 2023-11-15T04:29:39.621 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.616+0000 7f7e16ffd700 1 -- 172.21.15.190:0/4094841727 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e180095f0 con 0x7f7e280f51a0 2023-11-15T04:29:39.622 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.616+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e28059100 con 0x7f7e280f51a0 2023-11-15T04:29:39.622 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.616+0000 7f7e16ffd700 1 --2- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7e1002ff60 0x7f7e10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:39.622 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.616+0000 7f7e16ffd700 1 -- 172.21.15.190:0/4094841727 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e18041000 con 0x7f7e280f51a0 2023-11-15T04:29:39.622 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.617+0000 7f7e25d9b700 1 --2- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7e1002ff60 0x7f7e10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:39.622 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.617+0000 7f7e25d9b700 1 --2- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7e1002ff60 0x7f7e10032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7e1c0096a0 tx=0x7f7e1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:39.623 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.621+0000 7f7e16ffd700 1 -- 172.21.15.190:0/4094841727 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e1801c790 con 0x7f7e280f51a0 2023-11-15T04:29:39.798 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.797+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 --> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi047:172.21.15.47=smithi047;smithi190:172.21.15.190=smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e280f27b0 con 0x7f7e1002ff60 2023-11-15T04:29:39.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.803+0000 7f7e16ffd700 1 -- 172.21.15.190:0/4094841727 <== mgr.14164 v2:172.21.15.47:6800/1025769174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7e280f27b0 con 0x7f7e1002ff60 2023-11-15T04:29:39.804 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mon update... 2023-11-15T04:29:39.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7e1002ff60 msgr2=0x7f7e10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:39.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 --2- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7e1002ff60 0x7f7e10032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7e1c0096a0 tx=0x7f7e1c006b40).stop 2023-11-15T04:29:39.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 msgr2=0x7f7e280fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:39.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 --2- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 0x7f7e280fd120 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f7e18013510 tx=0x7f7e1800b040).stop 2023-11-15T04:29:39.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 shutdown_connections 2023-11-15T04:29:39.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 --2- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7e1002ff60 0x7f7e10032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:39.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 --2- 172.21.15.190:0/4094841727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e280f51a0 0x7f7e280fd120 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:39.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 >> 172.21.15.190:0/4094841727 conn(0x7f7e280f09a0 msgr2=0x7f7e2805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:39.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 shutdown_connections 2023-11-15T04:29:39.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:39.805+0000 7f7e2759e700 1 -- 172.21.15.190:0/4094841727 wait complete. 2023-11-15T04:29:40.151 DEBUG:teuthology.orchestra.run.smithi190:mon.smithi190> sudo journalctl -f -n 0 -u ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi190.service 2023-11-15T04:29:40.154 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:29:40.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:29:40.182 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:-- Logs begin at Wed 2023-11-15 04:15:01 UTC. -- 2023-11-15T04:29:40.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:39.057973+0000 mgr.smithi047.zgysuz (mgr.14164) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:40.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:39.804437+0000 mon.smithi047 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:29:41.274 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.272+0000 7fe70654c700 1 -- 172.21.15.190:0/1980854915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 msgr2=0x7fe7000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:41.274 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.272+0000 7fe70654c700 1 --2- 172.21.15.190:0/1980854915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 0x7fe7000f55c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fe6fc009fb0 tx=0x7fe6fc009b30).stop 2023-11-15T04:29:41.274 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.272+0000 7fe70654c700 1 -- 172.21.15.190:0/1980854915 shutdown_connections 2023-11-15T04:29:41.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.272+0000 7fe70654c700 1 --2- 172.21.15.190:0/1980854915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 0x7fe7000f55c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:41.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.272+0000 7fe70654c700 1 -- 172.21.15.190:0/1980854915 >> 172.21.15.190:0/1980854915 conn(0x7fe7000f09a0 msgr2=0x7fe7000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:41.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.272+0000 7fe70654c700 1 -- 172.21.15.190:0/1980854915 shutdown_connections 2023-11-15T04:29:41.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.272+0000 7fe70654c700 1 -- 172.21.15.190:0/1980854915 wait complete. 2023-11-15T04:29:41.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.273+0000 7fe70654c700 1 Processor -- start 2023-11-15T04:29:41.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.273+0000 7fe70654c700 1 -- start start 2023-11-15T04:29:41.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.273+0000 7fe70654c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 0x7fe7001091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:41.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.273+0000 7fe70654c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe700060240 con 0x7fe7000f51a0 2023-11-15T04:29:41.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.273+0000 7fe70554a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 0x7fe7001091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:41.277 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.273+0000 7fe70554a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 0x7fe7001091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:44226/0 (socket says 172.21.15.190:44226) 2023-11-15T04:29:41.277 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.273+0000 7fe70554a700 1 -- 172.21.15.190:0/3632931688 learned_addr learned my addr 172.21.15.190:0/3632931688 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:29:41.277 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.273+0000 7fe70554a700 1 -- 172.21.15.190:0/3632931688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc005040 con 0x7fe7000f51a0 2023-11-15T04:29:41.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.274+0000 7fe70554a700 1 --2- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 0x7fe7001091b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe6fc013510 tx=0x7fe6fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:41.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.274+0000 7fe6f5ffb700 1 -- 172.21.15.190:0/3632931688 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6fc009320 con 0x7fe7000f51a0 2023-11-15T04:29:41.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.274+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe700060440 con 0x7fe7000f51a0 2023-11-15T04:29:41.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.274+0000 7fe6f5ffb700 1 -- 172.21.15.190:0/3632931688 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6fc013f00 con 0x7fe7000f51a0 2023-11-15T04:29:41.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.274+0000 7fe6f5ffb700 1 -- 172.21.15.190:0/3632931688 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6fc014dc0 con 0x7fe7000f51a0 2023-11-15T04:29:41.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.274+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe700109860 con 0x7fe7000f51a0 2023-11-15T04:29:41.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.275+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe700045ad0 con 0x7fe7000f51a0 2023-11-15T04:29:41.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.275+0000 7fe6f5ffb700 1 -- 172.21.15.190:0/3632931688 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6fc0095f0 con 0x7fe7000f51a0 2023-11-15T04:29:41.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.275+0000 7fe6f5ffb700 1 --2- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe6ec02ff60 0x7fe6ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:41.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.275+0000 7fe6f5ffb700 1 -- 172.21.15.190:0/3632931688 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6fc0410e0 con 0x7fe7000f51a0 2023-11-15T04:29:41.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.275+0000 7fe704d49700 1 --2- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe6ec02ff60 0x7fe6ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:41.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.276+0000 7fe704d49700 1 --2- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe6ec02ff60 0x7fe6ec032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe6f00096a0 tx=0x7fe6f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:41.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.280+0000 7fe6f5ffb700 1 -- 172.21.15.190:0/3632931688 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6fc014420 con 0x7fe7000f51a0 2023-11-15T04:29:41.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.482+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe70010a070 con 0x7fe7000f51a0 2023-11-15T04:29:41.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.483+0000 7fe6f5ffb700 1 -- 172.21.15.190:0/3632931688 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe6fc01c920 con 0x7fe7000f51a0 2023-11-15T04:29:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:29:41.485 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:29:41.485 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.484+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe6ec02ff60 msgr2=0x7fe6ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:41.486 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.484+0000 7fe70654c700 1 --2- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe6ec02ff60 0x7fe6ec032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe6f00096a0 tx=0x7fe6f0006b40).stop 2023-11-15T04:29:41.486 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.484+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 msgr2=0x7fe7001091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:41.486 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.484+0000 7fe70654c700 1 --2- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 0x7fe7001091b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe6fc013510 tx=0x7fe6fc00b040).stop 2023-11-15T04:29:41.486 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.484+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 shutdown_connections 2023-11-15T04:29:41.487 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.484+0000 7fe70654c700 1 --2- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe6ec02ff60 0x7fe6ec032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:41.487 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.484+0000 7fe70654c700 1 --2- 172.21.15.190:0/3632931688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7000f51a0 0x7fe7001091b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:41.487 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.484+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 >> 172.21.15.190:0/3632931688 conn(0x7fe7000f09a0 msgr2=0x7fe7000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:41.487 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.485+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 shutdown_connections 2023-11-15T04:29:41.487 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:41.485+0000 7fe70654c700 1 -- 172.21.15.190:0/3632931688 wait complete. 2023-11-15T04:29:41.488 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:29:41.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:41 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:39.800134+0000 mgr.smithi047.zgysuz (mgr.14164) 17 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi047:172.21.15.47=smithi047;smithi190:172.21.15.190=smithi190", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:29:41.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:41 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:29:39.801634+0000 mgr.smithi047.zgysuz (mgr.14164) 18 : cephadm [INF] Saving service mon spec with placement smithi047:172.21.15.47=smithi047;smithi190:172.21.15.190=smithi190;count:2 2023-11-15T04:29:42.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:41.058263+0000 mgr.smithi047.zgysuz (mgr.14164) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:42.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:41.484517+0000 mon.smithi047 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.190:0/3632931688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:29:42.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:29:42.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:29:44.048 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.046+0000 7feb844ad700 1 -- 172.21.15.190:0/2915653057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 msgr2=0x7feb7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:44.048 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.046+0000 7feb844ad700 1 --2- 172.21.15.190:0/2915653057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 0x7feb7c0f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7feb6c009fb0 tx=0x7feb6c009b30).stop 2023-11-15T04:29:44.049 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.046+0000 7feb844ad700 1 -- 172.21.15.190:0/2915653057 shutdown_connections 2023-11-15T04:29:44.049 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.046+0000 7feb844ad700 1 --2- 172.21.15.190:0/2915653057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 0x7feb7c0f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:44.049 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.046+0000 7feb844ad700 1 -- 172.21.15.190:0/2915653057 >> 172.21.15.190:0/2915653057 conn(0x7feb7c0f0b70 msgr2=0x7feb7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:44.049 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.046+0000 7feb844ad700 1 -- 172.21.15.190:0/2915653057 shutdown_connections 2023-11-15T04:29:44.050 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.047+0000 7feb844ad700 1 -- 172.21.15.190:0/2915653057 wait complete. 2023-11-15T04:29:44.050 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.047+0000 7feb844ad700 1 Processor -- start 2023-11-15T04:29:44.050 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.047+0000 7feb844ad700 1 -- start start 2023-11-15T04:29:44.050 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.047+0000 7feb844ad700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 0x7feb7c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:44.050 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.047+0000 7feb844ad700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c106bd0 con 0x7feb7c0f5410 2023-11-15T04:29:44.051 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.047+0000 7feb82249700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 0x7feb7c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:44.051 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.047+0000 7feb82249700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 0x7feb7c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:44246/0 (socket says 172.21.15.190:44246) 2023-11-15T04:29:44.051 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.047+0000 7feb82249700 1 -- 172.21.15.190:0/2072201614 learned_addr learned my addr 172.21.15.190:0/2072201614 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:29:44.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.048+0000 7feb82249700 1 -- 172.21.15.190:0/2072201614 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c005040 con 0x7feb7c0f5410 2023-11-15T04:29:44.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.048+0000 7feb82249700 1 --2- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 0x7feb7c1066e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7feb6c013510 tx=0x7feb6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:44.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.048+0000 7feb72ffd700 1 -- 172.21.15.190:0/2072201614 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c009320 con 0x7feb7c0f5410 2023-11-15T04:29:44.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.048+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c106dd0 con 0x7feb7c0f5410 2023-11-15T04:29:44.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.048+0000 7feb72ffd700 1 -- 172.21.15.190:0/2072201614 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb6c013f00 con 0x7feb7c0f5410 2023-11-15T04:29:44.054 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.048+0000 7feb72ffd700 1 -- 172.21.15.190:0/2072201614 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c014d90 con 0x7feb7c0f5410 2023-11-15T04:29:44.054 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.048+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c107260 con 0x7feb7c0f5410 2023-11-15T04:29:44.054 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.049+0000 7feb72ffd700 1 -- 172.21.15.190:0/2072201614 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb6c0095f0 con 0x7feb7c0f5410 2023-11-15T04:29:44.054 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.049+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7c045ad0 con 0x7feb7c0f5410 2023-11-15T04:29:44.054 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.049+0000 7feb72ffd700 1 --2- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feb6802ff60 0x7feb68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:44.055 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.049+0000 7feb72ffd700 1 -- 172.21.15.190:0/2072201614 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb6c041000 con 0x7feb7c0f5410 2023-11-15T04:29:44.055 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.049+0000 7feb81a48700 1 --2- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feb6802ff60 0x7feb68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:44.055 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.050+0000 7feb81a48700 1 --2- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feb6802ff60 0x7feb68032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7feb780096a0 tx=0x7feb78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:44.056 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.054+0000 7feb72ffd700 1 -- 172.21.15.190:0/2072201614 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb6c01d7f0 con 0x7feb7c0f5410 2023-11-15T04:29:44.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.265+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb7c0fae60 con 0x7feb7c0f5410 2023-11-15T04:29:44.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.265+0000 7feb72ffd700 1 -- 172.21.15.190:0/2072201614 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb6c010230 con 0x7feb7c0f5410 2023-11-15T04:29:44.267 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:29:44.267 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:29:44.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feb6802ff60 msgr2=0x7feb68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:44.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 --2- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feb6802ff60 0x7feb68032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7feb780096a0 tx=0x7feb78006b40).stop 2023-11-15T04:29:44.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 msgr2=0x7feb7c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:44.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 --2- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 0x7feb7c1066e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7feb6c013510 tx=0x7feb6c00b040).stop 2023-11-15T04:29:44.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 shutdown_connections 2023-11-15T04:29:44.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 --2- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7feb6802ff60 0x7feb68032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:44.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 --2- 172.21.15.190:0/2072201614 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f5410 0x7feb7c1066e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:44.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 >> 172.21.15.190:0/2072201614 conn(0x7feb7c0f0b70 msgr2=0x7feb7c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:44.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 shutdown_connections 2023-11-15T04:29:44.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:44.267+0000 7feb844ad700 1 -- 172.21.15.190:0/2072201614 wait complete. 2023-11-15T04:29:44.270 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:29:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:43.058655+0000 mgr.smithi047.zgysuz (mgr.14164) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:45 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:44.266834+0000 mon.smithi047 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.190:0/2072201614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:29:45.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:29:45.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:29:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:45.058948+0000 mgr.smithi047.zgysuz (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:46.797 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.796+0000 7fca1db63700 1 -- 172.21.15.190:0/324912048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 msgr2=0x7fca180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:46.798 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.796+0000 7fca1db63700 1 --2- 172.21.15.190:0/324912048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 0x7fca180f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fca0c009fb0 tx=0x7fca0c009b30).stop 2023-11-15T04:29:46.798 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.796+0000 7fca1db63700 1 -- 172.21.15.190:0/324912048 shutdown_connections 2023-11-15T04:29:46.798 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.796+0000 7fca1db63700 1 --2- 172.21.15.190:0/324912048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 0x7fca180f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:46.798 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.796+0000 7fca1db63700 1 -- 172.21.15.190:0/324912048 >> 172.21.15.190:0/324912048 conn(0x7fca180f1c70 msgr2=0x7fca180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:46.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.796+0000 7fca1db63700 1 -- 172.21.15.190:0/324912048 shutdown_connections 2023-11-15T04:29:46.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.796+0000 7fca1db63700 1 -- 172.21.15.190:0/324912048 wait complete. 2023-11-15T04:29:46.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.797+0000 7fca1db63700 1 Processor -- start 2023-11-15T04:29:46.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.797+0000 7fca1db63700 1 -- start start 2023-11-15T04:29:46.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.797+0000 7fca1db63700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 0x7fca18106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:46.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.797+0000 7fca1db63700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca18106c20 con 0x7fca180f6510 2023-11-15T04:29:46.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.797+0000 7fca177fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 0x7fca18106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:46.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.797+0000 7fca177fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 0x7fca18106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:44272/0 (socket says 172.21.15.190:44272) 2023-11-15T04:29:46.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.797+0000 7fca177fe700 1 -- 172.21.15.190:0/4205809757 learned_addr learned my addr 172.21.15.190:0/4205809757 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:29:46.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.798+0000 7fca177fe700 1 -- 172.21.15.190:0/4205809757 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c005040 con 0x7fca180f6510 2023-11-15T04:29:46.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.798+0000 7fca177fe700 1 --2- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 0x7fca18106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fca0c014000 tx=0x7fca0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:46.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.798+0000 7fc9fffff700 1 -- 172.21.15.190:0/4205809757 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0c0095f0 con 0x7fca180f6510 2023-11-15T04:29:46.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.798+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca18106e20 con 0x7fca180f6510 2023-11-15T04:29:46.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.798+0000 7fc9fffff700 1 -- 172.21.15.190:0/4205809757 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca0c007e70 con 0x7fca180f6510 2023-11-15T04:29:46.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.798+0000 7fc9fffff700 1 -- 172.21.15.190:0/4205809757 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0c014ea0 con 0x7fca180f6510 2023-11-15T04:29:46.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.798+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca181072b0 con 0x7fca180f6510 2023-11-15T04:29:46.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.799+0000 7fc9fffff700 1 -- 172.21.15.190:0/4205809757 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca0c025070 con 0x7fca180f6510 2023-11-15T04:29:46.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.799+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca18045ad0 con 0x7fca180f6510 2023-11-15T04:29:46.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.799+0000 7fc9fffff700 1 --2- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca0002ff60 0x7fca00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:46.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.799+0000 7fc9fffff700 1 -- 172.21.15.190:0/4205809757 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca0c040970 con 0x7fca180f6510 2023-11-15T04:29:46.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.799+0000 7fca16ffd700 1 --2- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca0002ff60 0x7fca00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:46.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.800+0000 7fca16ffd700 1 --2- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca0002ff60 0x7fca00032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fca080096a0 tx=0x7fca08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:46.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:46.804+0000 7fc9fffff700 1 -- 172.21.15.190:0/4205809757 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca0c0079c0 con 0x7fca180f6510 2023-11-15T04:29:47.009 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.009+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca180fbf60 con 0x7fca180f6510 2023-11-15T04:29:47.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.009+0000 7fc9fffff700 1 -- 172.21.15.190:0/4205809757 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca0c00f300 con 0x7fca180f6510 2023-11-15T04:29:47.011 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:29:47.011 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:29:47.012 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca0002ff60 msgr2=0x7fca00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:47.012 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 --2- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca0002ff60 0x7fca00032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fca080096a0 tx=0x7fca08006b40).stop 2023-11-15T04:29:47.013 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 msgr2=0x7fca18106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:47.013 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 --2- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 0x7fca18106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fca0c014000 tx=0x7fca0c00b040).stop 2023-11-15T04:29:47.013 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 shutdown_connections 2023-11-15T04:29:47.013 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 --2- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca0002ff60 0x7fca00032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:47.013 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 --2- 172.21.15.190:0/4205809757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca180f6510 0x7fca18106730 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:47.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 >> 172.21.15.190:0/4205809757 conn(0x7fca180f1c70 msgr2=0x7fca180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:47.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 shutdown_connections 2023-11-15T04:29:47.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:47.011+0000 7fca1db63700 1 -- 172.21.15.190:0/4205809757 wait complete. 2023-11-15T04:29:47.014 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:29:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:47.010810+0000 mon.smithi047 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.190:0/4205809757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:29:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:47.059307+0000 mgr.smithi047.zgysuz (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:48.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:29:48.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:29:49.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.538+0000 7fa233ec1700 1 -- 172.21.15.190:0/460593426 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 msgr2=0x7fa22c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:49.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.538+0000 7fa233ec1700 1 --2- 172.21.15.190:0/460593426 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 0x7fa22c0f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa21c009fb0 tx=0x7fa21c009b30).stop 2023-11-15T04:29:49.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.538+0000 7fa233ec1700 1 -- 172.21.15.190:0/460593426 shutdown_connections 2023-11-15T04:29:49.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.538+0000 7fa233ec1700 1 --2- 172.21.15.190:0/460593426 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 0x7fa22c0f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:49.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.538+0000 7fa233ec1700 1 -- 172.21.15.190:0/460593426 >> 172.21.15.190:0/460593426 conn(0x7fa22c0f1c70 msgr2=0x7fa22c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:49.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.538+0000 7fa233ec1700 1 -- 172.21.15.190:0/460593426 shutdown_connections 2023-11-15T04:29:49.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.538+0000 7fa233ec1700 1 -- 172.21.15.190:0/460593426 wait complete. 2023-11-15T04:29:49.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.539+0000 7fa233ec1700 1 Processor -- start 2023-11-15T04:29:49.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.539+0000 7fa233ec1700 1 -- start start 2023-11-15T04:29:49.543 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.539+0000 7fa233ec1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 0x7fa22c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:49.543 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.539+0000 7fa233ec1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c106c20 con 0x7fa22c0f6510 2023-11-15T04:29:49.543 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.539+0000 7fa231c5d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 0x7fa22c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:49.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.539+0000 7fa231c5d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 0x7fa22c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:44286/0 (socket says 172.21.15.190:44286) 2023-11-15T04:29:49.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.539+0000 7fa231c5d700 1 -- 172.21.15.190:0/1116717830 learned_addr learned my addr 172.21.15.190:0/1116717830 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:29:49.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.539+0000 7fa231c5d700 1 -- 172.21.15.190:0/1116717830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c005040 con 0x7fa22c0f6510 2023-11-15T04:29:49.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.540+0000 7fa231c5d700 1 --2- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 0x7fa22c106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fa21c013510 tx=0x7fa21c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:49.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.540+0000 7fa2227fc700 1 -- 172.21.15.190:0/1116717830 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa21c009320 con 0x7fa22c0f6510 2023-11-15T04:29:49.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.540+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22c106e20 con 0x7fa22c0f6510 2023-11-15T04:29:49.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.540+0000 7fa2227fc700 1 -- 172.21.15.190:0/1116717830 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa21c013f00 con 0x7fa22c0f6510 2023-11-15T04:29:49.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.540+0000 7fa2227fc700 1 -- 172.21.15.190:0/1116717830 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa21c014d90 con 0x7fa22c0f6510 2023-11-15T04:29:49.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.540+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22c1072b0 con 0x7fa22c0f6510 2023-11-15T04:29:49.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.541+0000 7fa2227fc700 1 -- 172.21.15.190:0/1116717830 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa21c0263a0 con 0x7fa22c0f6510 2023-11-15T04:29:49.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.541+0000 7fa2227fc700 1 --2- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa21802ff60 0x7fa218032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:49.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.541+0000 7fa2227fc700 1 -- 172.21.15.190:0/1116717830 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa21c0410f0 con 0x7fa22c0f6510 2023-11-15T04:29:49.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.541+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22c045ad0 con 0x7fa22c0f6510 2023-11-15T04:29:49.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.541+0000 7fa23145c700 1 --2- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa21802ff60 0x7fa218032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:49.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.542+0000 7fa23145c700 1 --2- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa21802ff60 0x7fa218032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa2280096a0 tx=0x7fa228006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:49.547 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.546+0000 7fa2227fc700 1 -- 172.21.15.190:0/1116717830 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa21c012920 con 0x7fa22c0f6510 2023-11-15T04:29:49.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.752+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa22c0fc050 con 0x7fa22c0f6510 2023-11-15T04:29:49.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.753+0000 7fa2227fc700 1 -- 172.21.15.190:0/1116717830 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa21c02b020 con 0x7fa22c0f6510 2023-11-15T04:29:49.755 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:29:49.755 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:29:49.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa21802ff60 msgr2=0x7fa218032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:49.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 --2- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa21802ff60 0x7fa218032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa2280096a0 tx=0x7fa228006b40).stop 2023-11-15T04:29:49.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 msgr2=0x7fa22c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:49.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 --2- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 0x7fa22c106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fa21c013510 tx=0x7fa21c00b040).stop 2023-11-15T04:29:49.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 shutdown_connections 2023-11-15T04:29:49.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 --2- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa21802ff60 0x7fa218032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:49.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 --2- 172.21.15.190:0/1116717830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa22c0f6510 0x7fa22c106730 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:49.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 >> 172.21.15.190:0/1116717830 conn(0x7fa22c0f1c70 msgr2=0x7fa22c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:49.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 shutdown_connections 2023-11-15T04:29:49.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:49.755+0000 7fa233ec1700 1 -- 172.21.15.190:0/1116717830 wait complete. 2023-11-15T04:29:49.758 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:29:50.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:50 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:49.059569+0000 mgr.smithi047.zgysuz (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:50.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:50 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:49.754601+0000 mon.smithi047 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.190:0/1116717830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:29:51.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:29:51.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:29:52.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.258+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/2453232233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 msgr2=0x7fd4e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:52.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.258+0000 7fd4e8c9f700 1 --2- 172.21.15.190:0/2453232233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 0x7fd4e40f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fd4d8009fb0 tx=0x7fd4d8009b30).stop 2023-11-15T04:29:52.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.258+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/2453232233 shutdown_connections 2023-11-15T04:29:52.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.258+0000 7fd4e8c9f700 1 --2- 172.21.15.190:0/2453232233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 0x7fd4e40f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:52.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.258+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/2453232233 >> 172.21.15.190:0/2453232233 conn(0x7fd4e40f0b70 msgr2=0x7fd4e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:52.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.258+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/2453232233 shutdown_connections 2023-11-15T04:29:52.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.258+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/2453232233 wait complete. 2023-11-15T04:29:52.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.259+0000 7fd4e8c9f700 1 Processor -- start 2023-11-15T04:29:52.262 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.259+0000 7fd4e8c9f700 1 -- start start 2023-11-15T04:29:52.262 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.259+0000 7fd4e8c9f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 0x7fd4e4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:52.262 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.259+0000 7fd4e8c9f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e4060220 con 0x7fd4e40f5410 2023-11-15T04:29:52.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.259+0000 7fd4e259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 0x7fd4e4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:52.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.259+0000 7fd4e259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 0x7fd4e4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53676/0 (socket says 172.21.15.190:53676) 2023-11-15T04:29:52.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.259+0000 7fd4e259c700 1 -- 172.21.15.190:0/3319942230 learned_addr learned my addr 172.21.15.190:0/3319942230 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:29:52.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.260+0000 7fd4e259c700 1 -- 172.21.15.190:0/3319942230 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d8005040 con 0x7fd4e40f5410 2023-11-15T04:29:52.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.260+0000 7fd4e259c700 1 --2- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 0x7fd4e4109450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd4d8014000 tx=0x7fd4d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:52.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.260+0000 7fd4d2ffd700 1 -- 172.21.15.190:0/3319942230 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d80095f0 con 0x7fd4e40f5410 2023-11-15T04:29:52.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.260+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e4060420 con 0x7fd4e40f5410 2023-11-15T04:29:52.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.260+0000 7fd4d2ffd700 1 -- 172.21.15.190:0/3319942230 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4d8007e70 con 0x7fd4e40f5410 2023-11-15T04:29:52.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.260+0000 7fd4d2ffd700 1 -- 172.21.15.190:0/3319942230 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d8014ea0 con 0x7fd4e40f5410 2023-11-15T04:29:52.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.260+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e4109b00 con 0x7fd4e40f5410 2023-11-15T04:29:52.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.261+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e4059100 con 0x7fd4e40f5410 2023-11-15T04:29:52.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.261+0000 7fd4d2ffd700 1 -- 172.21.15.190:0/3319942230 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4d8025070 con 0x7fd4e40f5410 2023-11-15T04:29:52.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.261+0000 7fd4d2ffd700 1 --2- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd4cc02ffb0 0x7fd4cc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:52.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.261+0000 7fd4d2ffd700 1 -- 172.21.15.190:0/3319942230 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4d8040a00 con 0x7fd4e40f5410 2023-11-15T04:29:52.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.262+0000 7fd4e1d9b700 1 --2- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd4cc02ffb0 0x7fd4cc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:52.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.262+0000 7fd4e1d9b700 1 --2- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd4cc02ffb0 0x7fd4cc032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fd4d40096a0 tx=0x7fd4d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:52.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.267+0000 7fd4d2ffd700 1 -- 172.21.15.190:0/3319942230 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4d80079c0 con 0x7fd4e40f5410 2023-11-15T04:29:52.473 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.472+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4e4045ad0 con 0x7fd4e40f5410 2023-11-15T04:29:52.474 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.473+0000 7fd4d2ffd700 1 -- 172.21.15.190:0/3319942230 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd4d800f300 con 0x7fd4e40f5410 2023-11-15T04:29:52.474 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:29:52.475 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:29:52.475 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.474+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd4cc02ffb0 msgr2=0x7fd4cc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:52.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.474+0000 7fd4e8c9f700 1 --2- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd4cc02ffb0 0x7fd4cc032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fd4d40096a0 tx=0x7fd4d4006b40).stop 2023-11-15T04:29:52.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.474+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 msgr2=0x7fd4e4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:52.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.474+0000 7fd4e8c9f700 1 --2- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 0x7fd4e4109450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd4d8014000 tx=0x7fd4d800b040).stop 2023-11-15T04:29:52.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.474+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 shutdown_connections 2023-11-15T04:29:52.477 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.474+0000 7fd4e8c9f700 1 --2- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd4cc02ffb0 0x7fd4cc032470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:52.477 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.474+0000 7fd4e8c9f700 1 --2- 172.21.15.190:0/3319942230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e40f5410 0x7fd4e4109450 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:52.477 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.474+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 >> 172.21.15.190:0/3319942230 conn(0x7fd4e40f0b70 msgr2=0x7fd4e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:52.477 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.475+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 shutdown_connections 2023-11-15T04:29:52.477 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:52.475+0000 7fd4e8c9f700 1 -- 172.21.15.190:0/3319942230 wait complete. 2023-11-15T04:29:52.478 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:29:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:51.059912+0000 mgr.smithi047.zgysuz (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:53.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:52.474458+0000 mon.smithi047 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.190:0/3319942230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:29:53.861 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:29:53.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:29:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:53.060354+0000 mgr.smithi047.zgysuz (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:54.963 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.961+0000 7f88acadb700 1 -- 172.21.15.190:0/2470009232 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 msgr2=0x7f88a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:54.964 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.961+0000 7f88acadb700 1 --2- 172.21.15.190:0/2470009232 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 0x7f88a80f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8898009fb0 tx=0x7f8898009b30).stop 2023-11-15T04:29:54.964 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.961+0000 7f88acadb700 1 -- 172.21.15.190:0/2470009232 shutdown_connections 2023-11-15T04:29:54.964 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.961+0000 7f88acadb700 1 --2- 172.21.15.190:0/2470009232 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 0x7f88a80f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:54.964 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.961+0000 7f88acadb700 1 -- 172.21.15.190:0/2470009232 >> 172.21.15.190:0/2470009232 conn(0x7f88a80f1c70 msgr2=0x7f88a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:54.965 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.961+0000 7f88acadb700 1 -- 172.21.15.190:0/2470009232 shutdown_connections 2023-11-15T04:29:54.965 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.961+0000 7f88acadb700 1 -- 172.21.15.190:0/2470009232 wait complete. 2023-11-15T04:29:54.965 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.962+0000 7f88acadb700 1 Processor -- start 2023-11-15T04:29:54.965 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.962+0000 7f88acadb700 1 -- start start 2023-11-15T04:29:54.965 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.962+0000 7f88acadb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 0x7f88a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:54.966 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.962+0000 7f88acadb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a8106c20 con 0x7f88a80f6510 2023-11-15T04:29:54.966 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.962+0000 7f88a659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 0x7f88a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:54.966 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.962+0000 7f88a659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 0x7f88a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53688/0 (socket says 172.21.15.190:53688) 2023-11-15T04:29:54.966 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.962+0000 7f88a659c700 1 -- 172.21.15.190:0/30093458 learned_addr learned my addr 172.21.15.190:0/30093458 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:29:54.966 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.962+0000 7f88a659c700 1 -- 172.21.15.190:0/30093458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8898005040 con 0x7f88a80f6510 2023-11-15T04:29:54.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.962+0000 7f88a659c700 1 --2- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 0x7f88a8106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8898013510 tx=0x7f889800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:54.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.963+0000 7f889effd700 1 -- 172.21.15.190:0/30093458 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8898009320 con 0x7f88a80f6510 2023-11-15T04:29:54.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.963+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a8106e20 con 0x7f88a80f6510 2023-11-15T04:29:54.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.963+0000 7f889effd700 1 -- 172.21.15.190:0/30093458 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8898013f00 con 0x7f88a80f6510 2023-11-15T04:29:54.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.963+0000 7f889effd700 1 -- 172.21.15.190:0/30093458 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8898014d90 con 0x7f88a80f6510 2023-11-15T04:29:54.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.963+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a81072b0 con 0x7f88a80f6510 2023-11-15T04:29:54.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.963+0000 7f889effd700 1 -- 172.21.15.190:0/30093458 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88980095f0 con 0x7f88a80f6510 2023-11-15T04:29:54.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.964+0000 7f889effd700 1 --2- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f889402ff60 0x7f8894032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:29:54.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.964+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a8045ad0 con 0x7f88a80f6510 2023-11-15T04:29:54.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.964+0000 7f889effd700 1 -- 172.21.15.190:0/30093458 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88980410c0 con 0x7f88a80f6510 2023-11-15T04:29:54.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.964+0000 7f88a5d9b700 1 --2- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f889402ff60 0x7f8894032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:29:54.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.964+0000 7f88a5d9b700 1 --2- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f889402ff60 0x7f8894032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f88900096a0 tx=0x7f8890006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:29:54.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:54.969+0000 7f889effd700 1 -- 172.21.15.190:0/30093458 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8898014420 con 0x7f88a80f6510 2023-11-15T04:29:55.177 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.176+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88a80fc050 con 0x7f88a80f6510 2023-11-15T04:29:55.178 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.177+0000 7f889effd700 1 -- 172.21.15.190:0/30093458 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f889801d7f0 con 0x7f88a80f6510 2023-11-15T04:29:55.178 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:29:55.179 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:29:55.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.178+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f889402ff60 msgr2=0x7f8894032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:55.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.178+0000 7f88acadb700 1 --2- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f889402ff60 0x7f8894032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f88900096a0 tx=0x7f8890006b40).stop 2023-11-15T04:29:55.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.178+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 msgr2=0x7f88a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:29:55.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.178+0000 7f88acadb700 1 --2- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 0x7f88a8106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8898013510 tx=0x7f889800b040).stop 2023-11-15T04:29:55.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.178+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 shutdown_connections 2023-11-15T04:29:55.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.178+0000 7f88acadb700 1 --2- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f889402ff60 0x7f8894032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:55.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.178+0000 7f88acadb700 1 --2- 172.21.15.190:0/30093458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88a80f6510 0x7f88a8106730 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:29:55.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.178+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 >> 172.21.15.190:0/30093458 conn(0x7f88a80f1c70 msgr2=0x7f88a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:29:55.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.178+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 shutdown_connections 2023-11-15T04:29:55.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:29:55.179+0000 7f88acadb700 1 -- 172.21.15.190:0/30093458 wait complete. 2023-11-15T04:29:55.182 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:29:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:55.060665+0000 mgr.smithi047.zgysuz (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:29:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:29:55.178389+0000 mon.smithi047 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.190:0/30093458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:29:56.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:29:56.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:29:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:29:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:57.061059+0000 mgr.smithi047.zgysuz (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:29:59.061405+0000 mgr.smithi047.zgysuz (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:01.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.343+0000 7f8cce5fa700 1 -- 172.21.15.190:0/2954273962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 msgr2=0x7f8cc80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:01.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.343+0000 7f8cce5fa700 1 --2- 172.21.15.190:0/2954273962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 0x7f8cc80f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f8cb8009fb0 tx=0x7f8cb8009b30).stop 2023-11-15T04:30:01.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.343+0000 7f8cce5fa700 1 -- 172.21.15.190:0/2954273962 shutdown_connections 2023-11-15T04:30:01.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.343+0000 7f8cce5fa700 1 --2- 172.21.15.190:0/2954273962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 0x7f8cc80f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:01.431 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.343+0000 7f8cce5fa700 1 -- 172.21.15.190:0/2954273962 >> 172.21.15.190:0/2954273962 conn(0x7f8cc80f0b70 msgr2=0x7f8cc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:01.431 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.343+0000 7f8cce5fa700 1 -- 172.21.15.190:0/2954273962 shutdown_connections 2023-11-15T04:30:01.431 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.343+0000 7f8cce5fa700 1 -- 172.21.15.190:0/2954273962 wait complete. 2023-11-15T04:30:01.431 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.343+0000 7f8cce5fa700 1 Processor -- start 2023-11-15T04:30:01.431 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.344+0000 7f8cce5fa700 1 -- start start 2023-11-15T04:30:01.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.344+0000 7f8cce5fa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 0x7f8cc8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:01.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.344+0000 7f8cce5fa700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc8060220 con 0x7f8cc80f5410 2023-11-15T04:30:01.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.344+0000 7f8cc7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 0x7f8cc8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:01.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.344+0000 7f8cc7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 0x7f8cc8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60306/0 (socket says 172.21.15.190:60306) 2023-11-15T04:30:01.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.344+0000 7f8cc7fff700 1 -- 172.21.15.190:0/3479288792 learned_addr learned my addr 172.21.15.190:0/3479288792 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:01.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.345+0000 7f8cc7fff700 1 -- 172.21.15.190:0/3479288792 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb8005040 con 0x7f8cc80f5410 2023-11-15T04:30:01.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.345+0000 7f8cc7fff700 1 --2- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 0x7f8cc8109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8cb8014000 tx=0x7f8cb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:01.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.345+0000 7f8cc4ff9700 1 -- 172.21.15.190:0/3479288792 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cb80095f0 con 0x7f8cc80f5410 2023-11-15T04:30:01.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.345+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc8060420 con 0x7f8cc80f5410 2023-11-15T04:30:01.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.345+0000 7f8cc4ff9700 1 -- 172.21.15.190:0/3479288792 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cb8007e70 con 0x7f8cc80f5410 2023-11-15T04:30:01.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.345+0000 7f8cc4ff9700 1 -- 172.21.15.190:0/3479288792 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cb8014ea0 con 0x7f8cc80f5410 2023-11-15T04:30:01.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.345+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc8109b00 con 0x7f8cc80f5410 2023-11-15T04:30:01.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.346+0000 7f8cc4ff9700 1 -- 172.21.15.190:0/3479288792 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cb8025070 con 0x7f8cc80f5410 2023-11-15T04:30:01.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.346+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc8059100 con 0x7f8cc80f5410 2023-11-15T04:30:01.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.346+0000 7f8cc4ff9700 1 --2- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8cb002ff60 0x7f8cb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:01.435 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.346+0000 7f8cc4ff9700 1 -- 172.21.15.190:0/3479288792 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cb8040a20 con 0x7f8cc80f5410 2023-11-15T04:30:01.435 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.346+0000 7f8cc77fe700 1 --2- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8cb002ff60 0x7f8cb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:01.435 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.347+0000 7f8cc77fe700 1 --2- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8cb002ff60 0x7f8cb0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f8cbc0096a0 tx=0x7f8cbc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:01.435 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.351+0000 7f8cc4ff9700 1 -- 172.21.15.190:0/3479288792 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cb800f710 con 0x7f8cc80f5410 2023-11-15T04:30:01.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.558+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cc8045ad0 con 0x7f8cc80f5410 2023-11-15T04:30:01.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.559+0000 7f8cc4ff9700 1 -- 172.21.15.190:0/3479288792 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8cb8012330 con 0x7f8cc80f5410 2023-11-15T04:30:01.560 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:01.560 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:01.562 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8cb002ff60 msgr2=0x7f8cb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:01.562 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 --2- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8cb002ff60 0x7f8cb0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f8cbc0096a0 tx=0x7f8cbc006b40).stop 2023-11-15T04:30:01.562 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 msgr2=0x7f8cc8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:01.562 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 --2- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 0x7f8cc8109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8cb8014000 tx=0x7f8cb800b040).stop 2023-11-15T04:30:01.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 shutdown_connections 2023-11-15T04:30:01.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 --2- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8cb002ff60 0x7f8cb0032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:01.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 --2- 172.21.15.190:0/3479288792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8cc80f5410 0x7f8cc8109450 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:01.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 >> 172.21.15.190:0/3479288792 conn(0x7f8cc80f0b70 msgr2=0x7f8cc80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:01.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 shutdown_connections 2023-11-15T04:30:01.564 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:01.561+0000 7f8cce5fa700 1 -- 172.21.15.190:0/3479288792 wait complete. 2023-11-15T04:30:01.564 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:01.061780+0000 mgr.smithi047.zgysuz (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:01.560318+0000 mon.smithi047 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.190:0/3479288792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:03.062212+0000 mgr.smithi047.zgysuz (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:05.111 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:05.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:05.062570+0000 mgr.smithi047.zgysuz (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:05.213802+0000 mon.smithi047 (mon.0) 129 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:30:06.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:05.216871+0000 mon.smithi047 (mon.0) 130 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:30:07.661 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.659+0000 7f4294155700 1 -- 172.21.15.190:0/4102704791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 msgr2=0x7f428c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:07.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.659+0000 7f4294155700 1 --2- 172.21.15.190:0/4102704791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 0x7f428c0f6930 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4284009fb0 tx=0x7f4284009b30).stop 2023-11-15T04:30:07.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.659+0000 7f4294155700 1 -- 172.21.15.190:0/4102704791 shutdown_connections 2023-11-15T04:30:07.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.659+0000 7f4294155700 1 --2- 172.21.15.190:0/4102704791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 0x7f428c0f6930 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:07.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.659+0000 7f4294155700 1 -- 172.21.15.190:0/4102704791 >> 172.21.15.190:0/4102704791 conn(0x7f428c0f1c70 msgr2=0x7f428c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:07.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.659+0000 7f4294155700 1 -- 172.21.15.190:0/4102704791 shutdown_connections 2023-11-15T04:30:07.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.659+0000 7f4294155700 1 -- 172.21.15.190:0/4102704791 wait complete. 2023-11-15T04:30:07.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.660+0000 7f4294155700 1 Processor -- start 2023-11-15T04:30:07.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.660+0000 7f4294155700 1 -- start start 2023-11-15T04:30:07.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.660+0000 7f4294155700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 0x7f428c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:07.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.660+0000 7f4294155700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c069e20 con 0x7f428c0f6510 2023-11-15T04:30:07.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.660+0000 7f4291ef1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 0x7f428c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:07.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.660+0000 7f4291ef1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 0x7f428c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60324/0 (socket says 172.21.15.190:60324) 2023-11-15T04:30:07.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.660+0000 7f4291ef1700 1 -- 172.21.15.190:0/2497529750 learned_addr learned my addr 172.21.15.190:0/2497529750 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:07.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.660+0000 7f4291ef1700 1 -- 172.21.15.190:0/2497529750 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4284005040 con 0x7f428c0f6510 2023-11-15T04:30:07.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.661+0000 7f4291ef1700 1 --2- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 0x7f428c06b9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4284014000 tx=0x7f428400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:07.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.661+0000 7f42827fc700 1 -- 172.21.15.190:0/2497529750 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42840095f0 con 0x7f428c0f6510 2023-11-15T04:30:07.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.661+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c069fc0 con 0x7f428c0f6510 2023-11-15T04:30:07.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.661+0000 7f42827fc700 1 -- 172.21.15.190:0/2497529750 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4284007e70 con 0x7f428c0f6510 2023-11-15T04:30:07.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.661+0000 7f42827fc700 1 -- 172.21.15.190:0/2497529750 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4284014ea0 con 0x7f428c0f6510 2023-11-15T04:30:07.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.661+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c06a450 con 0x7f428c0f6510 2023-11-15T04:30:07.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.662+0000 7f42827fc700 1 -- 172.21.15.190:0/2497529750 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4284025070 con 0x7f428c0f6510 2023-11-15T04:30:07.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.662+0000 7f42827fc700 1 --2- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f427802ff60 0x7f4278032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:07.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.662+0000 7f42827fc700 1 -- 172.21.15.190:0/2497529750 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4284040930 con 0x7f428c0f6510 2023-11-15T04:30:07.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.662+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c059100 con 0x7f428c0f6510 2023-11-15T04:30:07.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.662+0000 7f42916f0700 1 --2- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f427802ff60 0x7f4278032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:07.668 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.662+0000 7f42916f0700 1 --2- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f427802ff60 0x7f4278032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f427c0096a0 tx=0x7f427c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:07.668 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.667+0000 7f42827fc700 1 -- 172.21.15.190:0/2497529750 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f428400f4e0 con 0x7f428c0f6510 2023-11-15T04:30:07.871 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.871+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f428c0fbf60 con 0x7f428c0f6510 2023-11-15T04:30:07.872 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.871+0000 7f42827fc700 1 -- 172.21.15.190:0/2497529750 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f428402d090 con 0x7f428c0f6510 2023-11-15T04:30:07.872 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:07.873 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:07.874 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f427802ff60 msgr2=0x7f4278032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 --2- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f427802ff60 0x7f4278032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f427c0096a0 tx=0x7f427c006b40).stop 2023-11-15T04:30:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 msgr2=0x7f428c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 --2- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 0x7f428c06b9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4284014000 tx=0x7f428400b040).stop 2023-11-15T04:30:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 shutdown_connections 2023-11-15T04:30:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 --2- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f427802ff60 0x7f4278032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:07.876 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 --2- 172.21.15.190:0/2497529750 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428c0f6510 0x7f428c06b9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:07.876 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 >> 172.21.15.190:0/2497529750 conn(0x7f428c0f1c70 msgr2=0x7f428c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:07.876 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 shutdown_connections 2023-11-15T04:30:07.876 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:07.873+0000 7f4294155700 1 -- 172.21.15.190:0/2497529750 wait complete. 2023-11-15T04:30:07.876 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:08.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:07.062839+0000 mgr.smithi047.zgysuz (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:09.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:07.872880+0000 mon.smithi047 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.190:0/2497529750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:09.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:09.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:10.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:09.063149+0000 mgr.smithi047.zgysuz (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:10.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.398+0000 7f9c92590700 1 -- 172.21.15.190:0/916981356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 msgr2=0x7f9c8c0f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:10.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.398+0000 7f9c92590700 1 --2- 172.21.15.190:0/916981356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 0x7f9c8c0f5bd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9c88009fb0 tx=0x7f9c88009b30).stop 2023-11-15T04:30:10.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.398+0000 7f9c92590700 1 -- 172.21.15.190:0/916981356 shutdown_connections 2023-11-15T04:30:10.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.398+0000 7f9c92590700 1 --2- 172.21.15.190:0/916981356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 0x7f9c8c0f5bd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:10.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.398+0000 7f9c92590700 1 -- 172.21.15.190:0/916981356 >> 172.21.15.190:0/916981356 conn(0x7f9c8c0f0120 msgr2=0x7f9c8c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:10.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.398+0000 7f9c92590700 1 -- 172.21.15.190:0/916981356 shutdown_connections 2023-11-15T04:30:10.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.398+0000 7f9c92590700 1 -- 172.21.15.190:0/916981356 wait complete. 2023-11-15T04:30:10.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.398+0000 7f9c92590700 1 Processor -- start 2023-11-15T04:30:10.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.399+0000 7f9c92590700 1 -- start start 2023-11-15T04:30:10.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.399+0000 7f9c92590700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 0x7f9c8c1053c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:10.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.399+0000 7f9c92590700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c1058b0 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.399+0000 7f9c9158e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 0x7f9c8c1053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:10.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.399+0000 7f9c9158e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 0x7f9c8c1053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41484/0 (socket says 172.21.15.190:41484) 2023-11-15T04:30:10.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.399+0000 7f9c9158e700 1 -- 172.21.15.190:0/2462671046 learned_addr learned my addr 172.21.15.190:0/2462671046 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:10.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.399+0000 7f9c9158e700 1 -- 172.21.15.190:0/2462671046 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c88005040 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.399+0000 7f9c9158e700 1 --2- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 0x7f9c8c1053c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9c88013510 tx=0x7f9c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:10.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.400+0000 7f9c81ffb700 1 -- 172.21.15.190:0/2462671046 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c88009320 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.400+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8c105ab0 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.400+0000 7f9c81ffb700 1 -- 172.21.15.190:0/2462671046 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c88013f00 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.400+0000 7f9c81ffb700 1 -- 172.21.15.190:0/2462671046 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c88014dc0 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.400+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8c105f40 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.401+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8c045ad0 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.401+0000 7f9c81ffb700 1 -- 172.21.15.190:0/2462671046 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c880095f0 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.401+0000 7f9c81ffb700 1 --2- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9c7802ff60 0x7f9c78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:10.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.401+0000 7f9c81ffb700 1 -- 172.21.15.190:0/2462671046 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c88041040 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.401+0000 7f9c90d8d700 1 --2- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9c7802ff60 0x7f9c78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:10.406 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.402+0000 7f9c90d8d700 1 --2- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9c7802ff60 0x7f9c78032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9c7c0096a0 tx=0x7f9c7c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:10.406 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.406+0000 7f9c81ffb700 1 -- 172.21.15.190:0/2462671046 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c88014420 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.611+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c8c106df0 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.613 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.612+0000 7f9c81ffb700 1 -- 172.21.15.190:0/2462671046 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c88010230 con 0x7f9c8c0f57b0 2023-11-15T04:30:10.613 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:10.614 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:10.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9c7802ff60 msgr2=0x7f9c78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:10.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 --2- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9c7802ff60 0x7f9c78032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9c7c0096a0 tx=0x7f9c7c006b40).stop 2023-11-15T04:30:10.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 msgr2=0x7f9c8c1053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:10.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 --2- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 0x7f9c8c1053c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9c88013510 tx=0x7f9c8800b040).stop 2023-11-15T04:30:10.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 shutdown_connections 2023-11-15T04:30:10.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 --2- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9c7802ff60 0x7f9c78032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:10.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 --2- 172.21.15.190:0/2462671046 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c8c0f57b0 0x7f9c8c1053c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:10.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 >> 172.21.15.190:0/2462671046 conn(0x7f9c8c0f0120 msgr2=0x7f9c8c0f0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:10.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 shutdown_connections 2023-11-15T04:30:10.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:10.614+0000 7f9c92590700 1 -- 172.21.15.190:0/2462671046 wait complete. 2023-11-15T04:30:10.618 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:11.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:10.613451+0000 mon.smithi047 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.190:0/2462671046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:11.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:11.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:12.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:11.063443+0000 mgr.smithi047.zgysuz (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:13.081 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.080+0000 7fc411694700 1 -- 172.21.15.190:0/680611357 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c060280 msgr2=0x7fc40c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:13.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.080+0000 7fc411694700 1 --2- 172.21.15.190:0/680611357 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c060280 0x7fc40c0f90f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fc400009fb0 tx=0x7fc400009b30).stop 2023-11-15T04:30:13.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.080+0000 7fc411694700 1 -- 172.21.15.190:0/680611357 shutdown_connections 2023-11-15T04:30:13.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.080+0000 7fc411694700 1 --2- 172.21.15.190:0/680611357 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c060280 0x7fc40c0f90f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:13.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.080+0000 7fc411694700 1 -- 172.21.15.190:0/680611357 >> 172.21.15.190:0/680611357 conn(0x7fc40c0f0b70 msgr2=0x7fc40c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:13.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.080+0000 7fc411694700 1 -- 172.21.15.190:0/680611357 shutdown_connections 2023-11-15T04:30:13.083 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.080+0000 7fc411694700 1 -- 172.21.15.190:0/680611357 wait complete. 2023-11-15T04:30:13.085 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.081+0000 7fc411694700 1 Processor -- start 2023-11-15T04:30:13.085 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.081+0000 7fc411694700 1 -- start start 2023-11-15T04:30:13.085 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.081+0000 7fc411694700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c1067b0 0x7fc40c106bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:13.085 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.081+0000 7fc411694700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c1070c0 con 0x7fc40c1067b0 2023-11-15T04:30:13.085 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.081+0000 7fc40affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c1067b0 0x7fc40c106bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:13.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.081+0000 7fc40affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c1067b0 0x7fc40c106bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41494/0 (socket says 172.21.15.190:41494) 2023-11-15T04:30:13.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.081+0000 7fc40affd700 1 -- 172.21.15.190:0/251490261 learned_addr learned my addr 172.21.15.190:0/251490261 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:13.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.082+0000 7fc40affd700 1 -- 172.21.15.190:0/251490261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400005040 con 0x7fc40c1067b0 2023-11-15T04:30:13.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.082+0000 7fc40affd700 1 --2- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c1067b0 0x7fc40c106bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc400009f80 tx=0x7fc400009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:13.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.082+0000 7fc3f37fe700 1 -- 172.21.15.190:0/251490261 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4000136c0 con 0x7fc40c1067b0 2023-11-15T04:30:13.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.082+0000 7fc3f37fe700 1 -- 172.21.15.190:0/251490261 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4000095f0 con 0x7fc40c1067b0 2023-11-15T04:30:13.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.082+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40c1072c0 con 0x7fc40c1067b0 2023-11-15T04:30:13.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.082+0000 7fc3f37fe700 1 -- 172.21.15.190:0/251490261 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc400014420 con 0x7fc40c1067b0 2023-11-15T04:30:13.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.082+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40c107e10 con 0x7fc40c1067b0 2023-11-15T04:30:13.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.083+0000 7fc3f37fe700 1 -- 172.21.15.190:0/251490261 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc400025060 con 0x7fc40c1067b0 2023-11-15T04:30:13.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.083+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40c059100 con 0x7fc40c1067b0 2023-11-15T04:30:13.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.083+0000 7fc3f37fe700 1 --2- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc3f402ff60 0x7fc3f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:13.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.083+0000 7fc3f37fe700 1 -- 172.21.15.190:0/251490261 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc400040d80 con 0x7fc40c1067b0 2023-11-15T04:30:13.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.083+0000 7fc40a7fc700 1 --2- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc3f402ff60 0x7fc3f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:13.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.084+0000 7fc40a7fc700 1 --2- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc3f402ff60 0x7fc3f4032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fc3fc0096a0 tx=0x7fc3fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:13.089 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.088+0000 7fc3f37fe700 1 -- 172.21.15.190:0/251490261 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc400004490 con 0x7fc40c1067b0 2023-11-15T04:30:13.294 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.294+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc40c045ad0 con 0x7fc40c1067b0 2023-11-15T04:30:13.295 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.294+0000 7fc3f37fe700 1 -- 172.21.15.190:0/251490261 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc400042090 con 0x7fc40c1067b0 2023-11-15T04:30:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:13.296 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:13.297 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc3f402ff60 msgr2=0x7fc3f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:13.297 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 --2- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc3f402ff60 0x7fc3f4032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fc3fc0096a0 tx=0x7fc3fc006b40).stop 2023-11-15T04:30:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c1067b0 msgr2=0x7fc40c106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 --2- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c1067b0 0x7fc40c106bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc400009f80 tx=0x7fc400009b30).stop 2023-11-15T04:30:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 shutdown_connections 2023-11-15T04:30:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 --2- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc3f402ff60 0x7fc3f4032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:13.299 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 --2- 172.21.15.190:0/251490261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc40c1067b0 0x7fc40c106bd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:13.299 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 >> 172.21.15.190:0/251490261 conn(0x7fc40c0f0b70 msgr2=0x7fc40c0f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:13.299 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 shutdown_connections 2023-11-15T04:30:13.299 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:13.296+0000 7fc411694700 1 -- 172.21.15.190:0/251490261 wait complete. 2023-11-15T04:30:13.299 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:13.063694+0000 mgr.smithi047.zgysuz (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:13.295767+0000 mon.smithi047 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.190:0/251490261' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:14.708 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:14.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:15.777 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.775+0000 7f6325b9f700 1 -- 172.21.15.190:0/332355592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 msgr2=0x7f63200f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:15.777 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.775+0000 7f6325b9f700 1 --2- 172.21.15.190:0/332355592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 0x7f63200f90f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f6308009fb0 tx=0x7f6308009b30).stop 2023-11-15T04:30:15.778 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.775+0000 7f6325b9f700 1 -- 172.21.15.190:0/332355592 shutdown_connections 2023-11-15T04:30:15.778 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.775+0000 7f6325b9f700 1 --2- 172.21.15.190:0/332355592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 0x7f63200f90f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:15.778 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.775+0000 7f6325b9f700 1 -- 172.21.15.190:0/332355592 >> 172.21.15.190:0/332355592 conn(0x7f63200f0b70 msgr2=0x7f63200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:15.778 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.775+0000 7f6325b9f700 1 -- 172.21.15.190:0/332355592 shutdown_connections 2023-11-15T04:30:15.778 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.775+0000 7f6325b9f700 1 -- 172.21.15.190:0/332355592 wait complete. 2023-11-15T04:30:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.775+0000 7f6325b9f700 1 Processor -- start 2023-11-15T04:30:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.776+0000 7f6325b9f700 1 -- start start 2023-11-15T04:30:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.776+0000 7f6325b9f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 0x7f63201066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.776+0000 7f6325b9f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6320106bd0 con 0x7f6320060280 2023-11-15T04:30:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.776+0000 7f631f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 0x7f63201066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.776+0000 7f631f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 0x7f63201066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41518/0 (socket says 172.21.15.190:41518) 2023-11-15T04:30:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.776+0000 7f631f7fe700 1 -- 172.21.15.190:0/119311650 learned_addr learned my addr 172.21.15.190:0/119311650 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.776+0000 7f631f7fe700 1 -- 172.21.15.190:0/119311650 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6308005040 con 0x7f6320060280 2023-11-15T04:30:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.776+0000 7f631f7fe700 1 --2- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 0x7f63201066e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6308013510 tx=0x7f630800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:15.781 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.777+0000 7f6317fff700 1 -- 172.21.15.190:0/119311650 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6308009320 con 0x7f6320060280 2023-11-15T04:30:15.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.777+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6320106dd0 con 0x7f6320060280 2023-11-15T04:30:15.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.777+0000 7f6317fff700 1 -- 172.21.15.190:0/119311650 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6308013f00 con 0x7f6320060280 2023-11-15T04:30:15.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.777+0000 7f6317fff700 1 -- 172.21.15.190:0/119311650 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6308014dc0 con 0x7f6320060280 2023-11-15T04:30:15.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.777+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6320107260 con 0x7f6320060280 2023-11-15T04:30:15.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.777+0000 7f6317fff700 1 -- 172.21.15.190:0/119311650 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63080095f0 con 0x7f6320060280 2023-11-15T04:30:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.777+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6320045ad0 con 0x7f6320060280 2023-11-15T04:30:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.778+0000 7f6317fff700 1 --2- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f630c02ff60 0x7f630c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.778+0000 7f6317fff700 1 -- 172.21.15.190:0/119311650 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6308041080 con 0x7f6320060280 2023-11-15T04:30:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.778+0000 7f631effd700 1 --2- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f630c02ff60 0x7f630c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.778+0000 7f631effd700 1 --2- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f630c02ff60 0x7f630c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f63100096a0 tx=0x7f6310006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.783+0000 7f6317fff700 1 -- 172.21.15.190:0/119311650 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f630801c790 con 0x7f6320060280 2023-11-15T04:30:15.989 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.989+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6320108110 con 0x7f6320060280 2023-11-15T04:30:15.990 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.989+0000 7f6317fff700 1 -- 172.21.15.190:0/119311650 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6308010230 con 0x7f6320060280 2023-11-15T04:30:15.991 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:15.991 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:15.992 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f630c02ff60 msgr2=0x7f630c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:15.992 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 --2- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f630c02ff60 0x7f630c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f63100096a0 tx=0x7f6310006b40).stop 2023-11-15T04:30:15.992 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 msgr2=0x7f63201066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:15.992 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 --2- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 0x7f63201066e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6308013510 tx=0x7f630800b040).stop 2023-11-15T04:30:15.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 shutdown_connections 2023-11-15T04:30:15.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 --2- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f630c02ff60 0x7f630c032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:15.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 --2- 172.21.15.190:0/119311650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6320060280 0x7f63201066e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:15.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 >> 172.21.15.190:0/119311650 conn(0x7f63200f0b70 msgr2=0x7f63200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:15.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 shutdown_connections 2023-11-15T04:30:15.994 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:15.991+0000 7f6325b9f700 1 -- 172.21.15.190:0/119311650 wait complete. 2023-11-15T04:30:15.994 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:15.064035+0000 mgr.smithi047.zgysuz (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:15.990868+0000 mon.smithi047 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.190:0/119311650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:17.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:17.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:18.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:17.064443+0000 mgr.smithi047.zgysuz (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:18.475 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.473+0000 7ffb6cab4700 1 -- 172.21.15.190:0/2657434253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 msgr2=0x7ffb680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:18.475 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.473+0000 7ffb6cab4700 1 --2- 172.21.15.190:0/2657434253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 0x7ffb680f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7ffb50009fb0 tx=0x7ffb50009b30).stop 2023-11-15T04:30:18.475 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.473+0000 7ffb6cab4700 1 -- 172.21.15.190:0/2657434253 shutdown_connections 2023-11-15T04:30:18.475 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.473+0000 7ffb6cab4700 1 --2- 172.21.15.190:0/2657434253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 0x7ffb680f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:18.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.473+0000 7ffb6cab4700 1 -- 172.21.15.190:0/2657434253 >> 172.21.15.190:0/2657434253 conn(0x7ffb680f0b70 msgr2=0x7ffb680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:18.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.473+0000 7ffb6cab4700 1 -- 172.21.15.190:0/2657434253 shutdown_connections 2023-11-15T04:30:18.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.473+0000 7ffb6cab4700 1 -- 172.21.15.190:0/2657434253 wait complete. 2023-11-15T04:30:18.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.474+0000 7ffb6cab4700 1 Processor -- start 2023-11-15T04:30:18.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.474+0000 7ffb6cab4700 1 -- start start 2023-11-15T04:30:18.477 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.474+0000 7ffb6cab4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 0x7ffb680fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:18.477 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.474+0000 7ffb6cab4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680fb820 con 0x7ffb680f5410 2023-11-15T04:30:18.478 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.474+0000 7ffb6659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 0x7ffb680fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:18.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.474+0000 7ffb6659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 0x7ffb680fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41538/0 (socket says 172.21.15.190:41538) 2023-11-15T04:30:18.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.474+0000 7ffb6659c700 1 -- 172.21.15.190:0/1992351912 learned_addr learned my addr 172.21.15.190:0/1992351912 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:18.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.474+0000 7ffb6659c700 1 -- 172.21.15.190:0/1992351912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50005040 con 0x7ffb680f5410 2023-11-15T04:30:18.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.475+0000 7ffb6659c700 1 --2- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 0x7ffb680fd3a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ffb50013510 tx=0x7ffb5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:18.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.475+0000 7ffb5effd700 1 -- 172.21.15.190:0/1992351912 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb50009320 con 0x7ffb680f5410 2023-11-15T04:30:18.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.475+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb680fb9c0 con 0x7ffb680f5410 2023-11-15T04:30:18.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.475+0000 7ffb5effd700 1 -- 172.21.15.190:0/1992351912 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb50013f00 con 0x7ffb680f5410 2023-11-15T04:30:18.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.475+0000 7ffb5effd700 1 -- 172.21.15.190:0/1992351912 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb50014d90 con 0x7ffb680f5410 2023-11-15T04:30:18.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.475+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb680fbe50 con 0x7ffb680f5410 2023-11-15T04:30:18.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.476+0000 7ffb5effd700 1 -- 172.21.15.190:0/1992351912 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb500095f0 con 0x7ffb680f5410 2023-11-15T04:30:18.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.476+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb68059100 con 0x7ffb680f5410 2023-11-15T04:30:18.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.476+0000 7ffb5effd700 1 --2- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ffb5402ff60 0x7ffb54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:18.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.476+0000 7ffb5effd700 1 -- 172.21.15.190:0/1992351912 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb50041000 con 0x7ffb680f5410 2023-11-15T04:30:18.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.476+0000 7ffb65d9b700 1 --2- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ffb5402ff60 0x7ffb54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:18.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.476+0000 7ffb65d9b700 1 --2- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ffb5402ff60 0x7ffb54032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7ffb580096a0 tx=0x7ffb58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:18.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.481+0000 7ffb5effd700 1 -- 172.21.15.190:0/1992351912 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb50014420 con 0x7ffb680f5410 2023-11-15T04:30:18.687 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.685+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb680fae60 con 0x7ffb680f5410 2023-11-15T04:30:18.687 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.686+0000 7ffb5effd700 1 -- 172.21.15.190:0/1992351912 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb50010230 con 0x7ffb680f5410 2023-11-15T04:30:18.688 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:18.688 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:18.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ffb5402ff60 msgr2=0x7ffb54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:18.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 --2- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ffb5402ff60 0x7ffb54032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7ffb580096a0 tx=0x7ffb58006b40).stop 2023-11-15T04:30:18.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 msgr2=0x7ffb680fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:18.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 --2- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 0x7ffb680fd3a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ffb50013510 tx=0x7ffb5000b040).stop 2023-11-15T04:30:18.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 shutdown_connections 2023-11-15T04:30:18.690 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 --2- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ffb5402ff60 0x7ffb54032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:18.690 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 --2- 172.21.15.190:0/1992351912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb680f5410 0x7ffb680fd3a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:18.690 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 >> 172.21.15.190:0/1992351912 conn(0x7ffb680f0b70 msgr2=0x7ffb6805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:18.690 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 shutdown_connections 2023-11-15T04:30:18.691 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:18.688+0000 7ffb6cab4700 1 -- 172.21.15.190:0/1992351912 wait complete. 2023-11-15T04:30:18.691 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:19.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:18.687513+0000 mon.smithi047 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.190:0/1992351912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:20.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:19 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:19.064784+0000 mgr.smithi047.zgysuz (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:20.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:20.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:21.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.179+0000 7fb1e9d03700 1 -- 172.21.15.190:0/1560034829 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 msgr2=0x7fb1e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:21.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.179+0000 7fb1e9d03700 1 --2- 172.21.15.190:0/1560034829 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e40f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb1d8009fb0 tx=0x7fb1d8009b30).stop 2023-11-15T04:30:21.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.179+0000 7fb1e9d03700 1 -- 172.21.15.190:0/1560034829 shutdown_connections 2023-11-15T04:30:21.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.179+0000 7fb1e9d03700 1 --2- 172.21.15.190:0/1560034829 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e40f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:21.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.179+0000 7fb1e9d03700 1 -- 172.21.15.190:0/1560034829 >> 172.21.15.190:0/1560034829 conn(0x7fb1e40f0b70 msgr2=0x7fb1e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:21.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.179+0000 7fb1e9d03700 1 -- 172.21.15.190:0/1560034829 shutdown_connections 2023-11-15T04:30:21.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.179+0000 7fb1e9d03700 1 -- 172.21.15.190:0/1560034829 wait complete. 2023-11-15T04:30:21.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.180+0000 7fb1e9d03700 1 Processor -- start 2023-11-15T04:30:21.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.180+0000 7fb1e9d03700 1 -- start start 2023-11-15T04:30:21.184 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.180+0000 7fb1e9d03700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:21.184 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.180+0000 7fb1e9d03700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e4106bd0 con 0x7fb1e40f5410 2023-11-15T04:30:21.184 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.180+0000 7fb1e37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:21.184 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.180+0000 7fb1e37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:46558/0 (socket says 172.21.15.190:46558) 2023-11-15T04:30:21.185 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.180+0000 7fb1e37fe700 1 -- 172.21.15.190:0/2771027047 learned_addr learned my addr 172.21.15.190:0/2771027047 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:21.185 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.181+0000 7fb1e37fe700 1 -- 172.21.15.190:0/2771027047 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d8005040 con 0x7fb1e40f5410 2023-11-15T04:30:21.185 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.181+0000 7fb1e37fe700 1 --2- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e41066e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb1d8013510 tx=0x7fb1d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:21.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.181+0000 7fb1cbfff700 1 -- 172.21.15.190:0/2771027047 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1d8009320 con 0x7fb1e40f5410 2023-11-15T04:30:21.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.181+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e4106dd0 con 0x7fb1e40f5410 2023-11-15T04:30:21.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.181+0000 7fb1cbfff700 1 -- 172.21.15.190:0/2771027047 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1d8013f00 con 0x7fb1e40f5410 2023-11-15T04:30:21.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.181+0000 7fb1cbfff700 1 -- 172.21.15.190:0/2771027047 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1d8014d90 con 0x7fb1e40f5410 2023-11-15T04:30:21.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.181+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e4107260 con 0x7fb1e40f5410 2023-11-15T04:30:21.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.182+0000 7fb1cbfff700 1 -- 172.21.15.190:0/2771027047 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1d80095f0 con 0x7fb1e40f5410 2023-11-15T04:30:21.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.182+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e4045ad0 con 0x7fb1e40f5410 2023-11-15T04:30:21.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.182+0000 7fb1cbfff700 1 --2- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb1cc02ff60 0x7fb1cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:21.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.182+0000 7fb1cbfff700 1 -- 172.21.15.190:0/2771027047 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1d8040fd0 con 0x7fb1e40f5410 2023-11-15T04:30:21.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.182+0000 7fb1e2ffd700 1 --2- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb1cc02ff60 0x7fb1cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:21.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.183+0000 7fb1e2ffd700 1 --2- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb1cc02ff60 0x7fb1cc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb1d40096a0 tx=0x7fb1d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:21.191 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.187+0000 7fb1cbfff700 1 -- 172.21.15.190:0/2771027047 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1d8014420 con 0x7fb1e40f5410 2023-11-15T04:30:21.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.396+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1e4108110 con 0x7fb1e40f5410 2023-11-15T04:30:21.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.397+0000 7fb1cbfff700 1 -- 172.21.15.190:0/2771027047 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb1d801d7f0 con 0x7fb1e40f5410 2023-11-15T04:30:21.399 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:21.399 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:21.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb1cc02ff60 msgr2=0x7fb1cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:21.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 --2- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb1cc02ff60 0x7fb1cc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb1d40096a0 tx=0x7fb1d4006b40).stop 2023-11-15T04:30:21.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 msgr2=0x7fb1e41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:21.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 --2- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e41066e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb1d8013510 tx=0x7fb1d800b040).stop 2023-11-15T04:30:21.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 shutdown_connections 2023-11-15T04:30:21.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 --2- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb1cc02ff60 0x7fb1cc032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:21.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 --2- 172.21.15.190:0/2771027047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e41066e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:21.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 >> 172.21.15.190:0/2771027047 conn(0x7fb1e40f0b70 msgr2=0x7fb1e405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:21.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 shutdown_connections 2023-11-15T04:30:21.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:21.398+0000 7fb1e9d03700 1 -- 172.21.15.190:0/2771027047 wait complete. 2023-11-15T04:30:21.402 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:22.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:21.065066+0000 mgr.smithi047.zgysuz (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:22.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:21.398146+0000 mon.smithi047 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.190:0/2771027047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:22.773 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:22.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:23.846 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.844+0000 7f8f4e753700 1 -- 172.21.15.190:0/2159903759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 msgr2=0x7f8f480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:23.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.844+0000 7f8f4e753700 1 --2- 172.21.15.190:0/2159903759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 0x7f8f480f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f8f38009fb0 tx=0x7f8f38009b30).stop 2023-11-15T04:30:23.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.844+0000 7f8f4e753700 1 -- 172.21.15.190:0/2159903759 shutdown_connections 2023-11-15T04:30:23.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.844+0000 7f8f4e753700 1 --2- 172.21.15.190:0/2159903759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 0x7f8f480f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:23.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.844+0000 7f8f4e753700 1 -- 172.21.15.190:0/2159903759 >> 172.21.15.190:0/2159903759 conn(0x7f8f480f1c70 msgr2=0x7f8f480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:23.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.845+0000 7f8f4e753700 1 -- 172.21.15.190:0/2159903759 shutdown_connections 2023-11-15T04:30:23.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.845+0000 7f8f4e753700 1 -- 172.21.15.190:0/2159903759 wait complete. 2023-11-15T04:30:23.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.845+0000 7f8f4e753700 1 Processor -- start 2023-11-15T04:30:23.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.845+0000 7f8f4e753700 1 -- start start 2023-11-15T04:30:23.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.846+0000 7f8f4e753700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 0x7f8f48106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:23.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.846+0000 7f8f4e753700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48106c20 con 0x7f8f480f6510 2023-11-15T04:30:23.851 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.846+0000 7f8f47fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 0x7f8f48106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:23.851 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.846+0000 7f8f47fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 0x7f8f48106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:46580/0 (socket says 172.21.15.190:46580) 2023-11-15T04:30:23.851 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.846+0000 7f8f47fff700 1 -- 172.21.15.190:0/3044702008 learned_addr learned my addr 172.21.15.190:0/3044702008 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:23.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.846+0000 7f8f47fff700 1 -- 172.21.15.190:0/3044702008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f38005040 con 0x7f8f480f6510 2023-11-15T04:30:23.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.846+0000 7f8f47fff700 1 --2- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 0x7f8f48106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8f38013510 tx=0x7f8f3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:23.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.847+0000 7f8f44ff9700 1 -- 172.21.15.190:0/3044702008 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f38009320 con 0x7f8f480f6510 2023-11-15T04:30:23.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.847+0000 7f8f44ff9700 1 -- 172.21.15.190:0/3044702008 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f38013f00 con 0x7f8f480f6510 2023-11-15T04:30:23.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.847+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f48106e20 con 0x7f8f480f6510 2023-11-15T04:30:23.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.847+0000 7f8f44ff9700 1 -- 172.21.15.190:0/3044702008 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f38014d90 con 0x7f8f480f6510 2023-11-15T04:30:23.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.847+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f481072b0 con 0x7f8f480f6510 2023-11-15T04:30:23.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.847+0000 7f8f44ff9700 1 -- 172.21.15.190:0/3044702008 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f380095f0 con 0x7f8f480f6510 2023-11-15T04:30:23.855 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.847+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f48045ad0 con 0x7f8f480f6510 2023-11-15T04:30:23.855 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.848+0000 7f8f44ff9700 1 --2- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f3402ff60 0x7f8f34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:23.855 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.848+0000 7f8f44ff9700 1 -- 172.21.15.190:0/3044702008 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f38040fe0 con 0x7f8f480f6510 2023-11-15T04:30:23.855 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.848+0000 7f8f477fe700 1 --2- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f3402ff60 0x7f8f34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:23.855 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.848+0000 7f8f477fe700 1 --2- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f3402ff60 0x7f8f34032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f8f300096a0 tx=0x7f8f30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:23.856 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:23.853+0000 7f8f44ff9700 1 -- 172.21.15.190:0/3044702008 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f3801d7f0 con 0x7f8f480f6510 2023-11-15T04:30:24.058 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.057+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f480fbf60 con 0x7f8f480f6510 2023-11-15T04:30:24.058 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.057+0000 7f8f44ff9700 1 -- 172.21.15.190:0/3044702008 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f38010230 con 0x7f8f480f6510 2023-11-15T04:30:24.059 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:24.059 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:24.061 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f3402ff60 msgr2=0x7f8f34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:24.061 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 --2- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f3402ff60 0x7f8f34032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f8f300096a0 tx=0x7f8f30006b40).stop 2023-11-15T04:30:24.061 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 msgr2=0x7f8f48106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:24.061 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 --2- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 0x7f8f48106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8f38013510 tx=0x7f8f3800b040).stop 2023-11-15T04:30:24.062 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 shutdown_connections 2023-11-15T04:30:24.062 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 --2- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f3402ff60 0x7f8f34032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:24.062 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 --2- 172.21.15.190:0/3044702008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f480f6510 0x7f8f48106730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:24.063 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 >> 172.21.15.190:0/3044702008 conn(0x7f8f480f1c70 msgr2=0x7f8f480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:24.063 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 shutdown_connections 2023-11-15T04:30:24.063 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:24.059+0000 7f8f4e753700 1 -- 172.21.15.190:0/3044702008 wait complete. 2023-11-15T04:30:24.063 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:24.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:23.065492+0000 mgr.smithi047.zgysuz (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:24.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:24.058862+0000 mon.smithi047 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.190:0/3044702008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:25.430 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:25.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:26.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:25.065927+0000 mgr.smithi047.zgysuz (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:26.534 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.529+0000 7ff836d23700 1 -- 172.21.15.190:0/3800954353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8300f5410 msgr2=0x7ff8300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:26.534 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.529+0000 7ff836d23700 1 --2- 172.21.15.190:0/3800954353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8300f5410 0x7ff8300f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7ff824009fb0 tx=0x7ff824009b30).stop 2023-11-15T04:30:26.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.529+0000 7ff836d23700 1 -- 172.21.15.190:0/3800954353 shutdown_connections 2023-11-15T04:30:26.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.529+0000 7ff836d23700 1 --2- 172.21.15.190:0/3800954353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8300f5410 0x7ff8300f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:26.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.529+0000 7ff836d23700 1 -- 172.21.15.190:0/3800954353 >> 172.21.15.190:0/3800954353 conn(0x7ff8300f0b70 msgr2=0x7ff8300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:26.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.529+0000 7ff836d23700 1 -- 172.21.15.190:0/3800954353 shutdown_connections 2023-11-15T04:30:26.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.529+0000 7ff836d23700 1 -- 172.21.15.190:0/3800954353 wait complete. 2023-11-15T04:30:26.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.530+0000 7ff836d23700 1 Processor -- start 2023-11-15T04:30:26.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.530+0000 7ff836d23700 1 -- start start 2023-11-15T04:30:26.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.530+0000 7ff836d23700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff830060240 0x7ff830109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:26.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.530+0000 7ff836d23700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff830109a30 con 0x7ff830060240 2023-11-15T04:30:26.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.530+0000 7ff834abf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff830060240 0x7ff830109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:26.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.530+0000 7ff834abf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff830060240 0x7ff830109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:46604/0 (socket says 172.21.15.190:46604) 2023-11-15T04:30:26.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.530+0000 7ff834abf700 1 -- 172.21.15.190:0/2224725238 learned_addr learned my addr 172.21.15.190:0/2224725238 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:26.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.531+0000 7ff834abf700 1 -- 172.21.15.190:0/2224725238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff824005040 con 0x7ff830060240 2023-11-15T04:30:26.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.531+0000 7ff834abf700 1 --2- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff830060240 0x7ff830109540 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff824009f80 tx=0x7ff824009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:26.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.531+0000 7ff82d7fa700 1 -- 172.21.15.190:0/2224725238 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8240136c0 con 0x7ff830060240 2023-11-15T04:30:26.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.531+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff830109b70 con 0x7ff830060240 2023-11-15T04:30:26.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.531+0000 7ff82d7fa700 1 -- 172.21.15.190:0/2224725238 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8240095f0 con 0x7ff830060240 2023-11-15T04:30:26.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.531+0000 7ff82d7fa700 1 -- 172.21.15.190:0/2224725238 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff824014420 con 0x7ff830060240 2023-11-15T04:30:26.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.531+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83010a000 con 0x7ff830060240 2023-11-15T04:30:26.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.532+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff830059100 con 0x7ff830060240 2023-11-15T04:30:26.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.532+0000 7ff82d7fa700 1 -- 172.21.15.190:0/2224725238 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff824025060 con 0x7ff830060240 2023-11-15T04:30:26.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.532+0000 7ff82d7fa700 1 --2- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff82002ff60 0x7ff820032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:26.543 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.532+0000 7ff82d7fa700 1 -- 172.21.15.190:0/2224725238 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff824040d80 con 0x7ff830060240 2023-11-15T04:30:26.543 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.532+0000 7ff82ffff700 1 --2- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff82002ff60 0x7ff820032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:26.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.533+0000 7ff82ffff700 1 --2- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff82002ff60 0x7ff820032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff81c0096a0 tx=0x7ff81c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:26.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.538+0000 7ff82d7fa700 1 -- 172.21.15.190:0/2224725238 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff82401cb60 con 0x7ff830060240 2023-11-15T04:30:26.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.744+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff830045ad0 con 0x7ff830060240 2023-11-15T04:30:26.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.744+0000 7ff82d7fa700 1 -- 172.21.15.190:0/2224725238 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff830045ad0 con 0x7ff830060240 2023-11-15T04:30:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:26.746 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:26.747 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff82002ff60 msgr2=0x7ff820032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:26.747 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 --2- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff82002ff60 0x7ff820032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff81c0096a0 tx=0x7ff81c006b40).stop 2023-11-15T04:30:26.747 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff830060240 msgr2=0x7ff830109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:26.748 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 --2- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff830060240 0x7ff830109540 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff824009f80 tx=0x7ff824009b30).stop 2023-11-15T04:30:26.748 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 shutdown_connections 2023-11-15T04:30:26.748 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 --2- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff82002ff60 0x7ff820032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:26.748 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 --2- 172.21.15.190:0/2224725238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff830060240 0x7ff830109540 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:26.748 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 >> 172.21.15.190:0/2224725238 conn(0x7ff8300f0b70 msgr2=0x7ff8300f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:26.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 shutdown_connections 2023-11-15T04:30:26.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:26.746+0000 7ff836d23700 1 -- 172.21.15.190:0/2224725238 wait complete. 2023-11-15T04:30:26.749 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:26.745877+0000 mon.smithi047 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.190:0/2224725238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:27.066288+0000 mgr.smithi047.zgysuz (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:28.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:28.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:29.233 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.231+0000 7fef794b9700 1 -- 172.21.15.190:0/315057228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 msgr2=0x7fef740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:29.233 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.231+0000 7fef794b9700 1 --2- 172.21.15.190:0/315057228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 0x7fef740f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fef64009fb0 tx=0x7fef64009b30).stop 2023-11-15T04:30:29.233 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.232+0000 7fef794b9700 1 -- 172.21.15.190:0/315057228 shutdown_connections 2023-11-15T04:30:29.234 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.232+0000 7fef794b9700 1 --2- 172.21.15.190:0/315057228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 0x7fef740f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:29.234 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.232+0000 7fef794b9700 1 -- 172.21.15.190:0/315057228 >> 172.21.15.190:0/315057228 conn(0x7fef740f0b70 msgr2=0x7fef740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:29.234 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.232+0000 7fef794b9700 1 -- 172.21.15.190:0/315057228 shutdown_connections 2023-11-15T04:30:29.234 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.232+0000 7fef794b9700 1 -- 172.21.15.190:0/315057228 wait complete. 2023-11-15T04:30:29.234 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.232+0000 7fef794b9700 1 Processor -- start 2023-11-15T04:30:29.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.232+0000 7fef794b9700 1 -- start start 2023-11-15T04:30:29.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.232+0000 7fef794b9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 0x7fef741088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:29.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.232+0000 7fef794b9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef74108dc0 con 0x7fef740f5410 2023-11-15T04:30:29.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.233+0000 7fef72ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 0x7fef741088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:29.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.233+0000 7fef72ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 0x7fef741088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:46622/0 (socket says 172.21.15.190:46622) 2023-11-15T04:30:29.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.233+0000 7fef72ffd700 1 -- 172.21.15.190:0/1539232154 learned_addr learned my addr 172.21.15.190:0/1539232154 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:29.236 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.233+0000 7fef72ffd700 1 -- 172.21.15.190:0/1539232154 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef64005040 con 0x7fef740f5410 2023-11-15T04:30:29.237 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.233+0000 7fef72ffd700 1 --2- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 0x7fef741088d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fef64013510 tx=0x7fef6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:29.237 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.234+0000 7fef5b7fe700 1 -- 172.21.15.190:0/1539232154 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef64009320 con 0x7fef740f5410 2023-11-15T04:30:29.238 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.234+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef74108fc0 con 0x7fef740f5410 2023-11-15T04:30:29.238 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.234+0000 7fef5b7fe700 1 -- 172.21.15.190:0/1539232154 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef64013f00 con 0x7fef740f5410 2023-11-15T04:30:29.238 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.234+0000 7fef5b7fe700 1 -- 172.21.15.190:0/1539232154 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef64014d90 con 0x7fef740f5410 2023-11-15T04:30:29.238 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.234+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef74109450 con 0x7fef740f5410 2023-11-15T04:30:29.238 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.234+0000 7fef5b7fe700 1 -- 172.21.15.190:0/1539232154 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef640263a0 con 0x7fef740f5410 2023-11-15T04:30:29.239 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.235+0000 7fef5b7fe700 1 --2- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fef5c02ff60 0x7fef5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:29.239 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.235+0000 7fef5b7fe700 1 -- 172.21.15.190:0/1539232154 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef64040ff0 con 0x7fef740f5410 2023-11-15T04:30:29.239 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.235+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef74045ad0 con 0x7fef740f5410 2023-11-15T04:30:29.239 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.235+0000 7fef727fc700 1 --2- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fef5c02ff60 0x7fef5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:29.239 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.235+0000 7fef727fc700 1 --2- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fef5c02ff60 0x7fef5c032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fef680096a0 tx=0x7fef68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:29.240 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.240+0000 7fef5b7fe700 1 -- 172.21.15.190:0/1539232154 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef64012920 con 0x7fef740f5410 2023-11-15T04:30:29.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.447+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef740faf50 con 0x7fef740f5410 2023-11-15T04:30:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.447+0000 7fef5b7fe700 1 -- 172.21.15.190:0/1539232154 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fef6402b020 con 0x7fef740f5410 2023-11-15T04:30:29.448 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:29.448 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fef5c02ff60 msgr2=0x7fef5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 --2- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fef5c02ff60 0x7fef5c032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fef680096a0 tx=0x7fef68006b40).stop 2023-11-15T04:30:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 msgr2=0x7fef741088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 --2- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 0x7fef741088d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fef64013510 tx=0x7fef6400b040).stop 2023-11-15T04:30:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 shutdown_connections 2023-11-15T04:30:29.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 --2- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fef5c02ff60 0x7fef5c032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:29.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 --2- 172.21.15.190:0/1539232154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef740f5410 0x7fef741088d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:29.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 >> 172.21.15.190:0/1539232154 conn(0x7fef740f0b70 msgr2=0x7fef7405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:29.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 shutdown_connections 2023-11-15T04:30:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:29.449+0000 7fef794b9700 1 -- 172.21.15.190:0/1539232154 wait complete. 2023-11-15T04:30:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:30.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:29.066726+0000 mgr.smithi047.zgysuz (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:30.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:29.447733+0000 mon.smithi047 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.190:0/1539232154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:30.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:30.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:31.943 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.942+0000 7f8126929700 1 -- 172.21.15.190:0/4036809100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 msgr2=0x7f81200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:31.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.942+0000 7f8126929700 1 --2- 172.21.15.190:0/4036809100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 0x7f81200f55c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f811c009fb0 tx=0x7f811c009b30).stop 2023-11-15T04:30:31.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.943+0000 7f8126929700 1 -- 172.21.15.190:0/4036809100 shutdown_connections 2023-11-15T04:30:31.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.943+0000 7f8126929700 1 --2- 172.21.15.190:0/4036809100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 0x7f81200f55c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:31.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.943+0000 7f8126929700 1 -- 172.21.15.190:0/4036809100 >> 172.21.15.190:0/4036809100 conn(0x7f81200f09a0 msgr2=0x7f81200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:31.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.943+0000 7f8126929700 1 -- 172.21.15.190:0/4036809100 shutdown_connections 2023-11-15T04:30:31.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.943+0000 7f8126929700 1 -- 172.21.15.190:0/4036809100 wait complete. 2023-11-15T04:30:31.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.943+0000 7f8126929700 1 Processor -- start 2023-11-15T04:30:31.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.943+0000 7f8126929700 1 -- start start 2023-11-15T04:30:31.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.943+0000 7f8126929700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 0x7f81201091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:31.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.943+0000 7f8126929700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8120060240 con 0x7f81200f51a0 2023-11-15T04:30:31.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f8125927700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 0x7f81201091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:31.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f8125927700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 0x7f81201091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34766/0 (socket says 172.21.15.190:34766) 2023-11-15T04:30:31.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f8125927700 1 -- 172.21.15.190:0/3485195919 learned_addr learned my addr 172.21.15.190:0/3485195919 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:31.947 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f8125927700 1 -- 172.21.15.190:0/3485195919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c005040 con 0x7f81200f51a0 2023-11-15T04:30:31.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f8125927700 1 --2- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 0x7f81201091b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f811c013510 tx=0x7f811c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:31.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f81127fc700 1 -- 172.21.15.190:0/3485195919 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f811c009320 con 0x7f81200f51a0 2023-11-15T04:30:31.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8120060440 con 0x7f81200f51a0 2023-11-15T04:30:31.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8120109860 con 0x7f81200f51a0 2023-11-15T04:30:31.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f81127fc700 1 -- 172.21.15.190:0/3485195919 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f811c013f00 con 0x7f81200f51a0 2023-11-15T04:30:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.944+0000 7f81127fc700 1 -- 172.21.15.190:0/3485195919 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f811c014d90 con 0x7f81200f51a0 2023-11-15T04:30:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.945+0000 7f81127fc700 1 -- 172.21.15.190:0/3485195919 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f811c0095f0 con 0x7f81200f51a0 2023-11-15T04:30:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.945+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8120045ad0 con 0x7f81200f51a0 2023-11-15T04:30:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.945+0000 7f81127fc700 1 --2- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f810c02ff60 0x7f810c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.945+0000 7f81127fc700 1 -- 172.21.15.190:0/3485195919 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f811c0410e0 con 0x7f81200f51a0 2023-11-15T04:30:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.945+0000 7f8125126700 1 --2- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f810c02ff60 0x7f810c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:31.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.946+0000 7f8125126700 1 --2- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f810c02ff60 0x7f810c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f81140096a0 tx=0x7f8114006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:31.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:31.950+0000 7f81127fc700 1 -- 172.21.15.190:0/3485195919 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f811c014420 con 0x7f81200f51a0 2023-11-15T04:30:32.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.156+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f812010a080 con 0x7f81200f51a0 2023-11-15T04:30:32.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.156+0000 7f81127fc700 1 -- 172.21.15.190:0/3485195919 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f811c012980 con 0x7f81200f51a0 2023-11-15T04:30:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:32.157 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:32.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f810c02ff60 msgr2=0x7f810c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:32.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 --2- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f810c02ff60 0x7f810c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f81140096a0 tx=0x7f8114006b40).stop 2023-11-15T04:30:32.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 msgr2=0x7f81201091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:32.159 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 --2- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 0x7f81201091b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f811c013510 tx=0x7f811c00b040).stop 2023-11-15T04:30:32.159 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 shutdown_connections 2023-11-15T04:30:32.159 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 --2- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f810c02ff60 0x7f810c032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:32.159 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 --2- 172.21.15.190:0/3485195919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81200f51a0 0x7f81201091b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:32.159 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 >> 172.21.15.190:0/3485195919 conn(0x7f81200f09a0 msgr2=0x7f81200f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:32.160 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 shutdown_connections 2023-11-15T04:30:32.160 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:32.158+0000 7f8126929700 1 -- 172.21.15.190:0/3485195919 wait complete. 2023-11-15T04:30:32.160 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:31.067084+0000 mgr.smithi047.zgysuz (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:33.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:33 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:32.156987+0000 mon.smithi047 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.190:0/3485195919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:33.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:33.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:34.509 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:33.067319+0000 mgr.smithi047.zgysuz (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:34.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.627+0000 7fb911579700 1 -- 172.21.15.190:0/831067935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 msgr2=0x7fb90c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:34.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.627+0000 7fb911579700 1 --2- 172.21.15.190:0/831067935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 0x7fb90c0f66c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb8fc009fb0 tx=0x7fb8fc009b30).stop 2023-11-15T04:30:34.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.627+0000 7fb911579700 1 -- 172.21.15.190:0/831067935 shutdown_connections 2023-11-15T04:30:34.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.627+0000 7fb911579700 1 --2- 172.21.15.190:0/831067935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 0x7fb90c0f66c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:34.629 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.627+0000 7fb911579700 1 -- 172.21.15.190:0/831067935 >> 172.21.15.190:0/831067935 conn(0x7fb90c0f1ac0 msgr2=0x7fb90c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:34.629 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.627+0000 7fb911579700 1 -- 172.21.15.190:0/831067935 shutdown_connections 2023-11-15T04:30:34.629 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.628+0000 7fb911579700 1 -- 172.21.15.190:0/831067935 wait complete. 2023-11-15T04:30:34.631 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.628+0000 7fb911579700 1 Processor -- start 2023-11-15T04:30:34.632 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.628+0000 7fb911579700 1 -- start start 2023-11-15T04:30:34.632 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.629+0000 7fb911579700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 0x7fb90c1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:34.632 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.629+0000 7fb911579700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c106990 con 0x7fb90c0f62a0 2023-11-15T04:30:34.633 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.629+0000 7fb90bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 0x7fb90c1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:34.633 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.629+0000 7fb90bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 0x7fb90c1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34782/0 (socket says 172.21.15.190:34782) 2023-11-15T04:30:34.633 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.629+0000 7fb90bfff700 1 -- 172.21.15.190:0/3511528319 learned_addr learned my addr 172.21.15.190:0/3511528319 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:34.633 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.629+0000 7fb90bfff700 1 -- 172.21.15.190:0/3511528319 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc005040 con 0x7fb90c0f62a0 2023-11-15T04:30:34.633 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.629+0000 7fb90bfff700 1 --2- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 0x7fb90c1064a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fb8fc013510 tx=0x7fb8fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:34.634 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.630+0000 7fb908ff9700 1 -- 172.21.15.190:0/3511528319 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8fc009320 con 0x7fb90c0f62a0 2023-11-15T04:30:34.634 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.630+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c106b90 con 0x7fb90c0f62a0 2023-11-15T04:30:34.635 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.630+0000 7fb908ff9700 1 -- 172.21.15.190:0/3511528319 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8fc013f00 con 0x7fb90c0f62a0 2023-11-15T04:30:34.636 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.630+0000 7fb908ff9700 1 -- 172.21.15.190:0/3511528319 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8fc014dc0 con 0x7fb90c0f62a0 2023-11-15T04:30:34.636 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.630+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c107020 con 0x7fb90c0f62a0 2023-11-15T04:30:34.636 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.630+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c045ad0 con 0x7fb90c0f62a0 2023-11-15T04:30:34.636 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.631+0000 7fb908ff9700 1 -- 172.21.15.190:0/3511528319 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8fc0095f0 con 0x7fb90c0f62a0 2023-11-15T04:30:34.636 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.631+0000 7fb908ff9700 1 --2- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8f402ff60 0x7fb8f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:34.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.631+0000 7fb908ff9700 1 -- 172.21.15.190:0/3511528319 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8fc0410f0 con 0x7fb90c0f62a0 2023-11-15T04:30:34.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.631+0000 7fb90b7fe700 1 --2- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8f402ff60 0x7fb8f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:34.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.632+0000 7fb90b7fe700 1 --2- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8f402ff60 0x7fb8f4032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb9000096a0 tx=0x7fb900006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:34.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.635+0000 7fb908ff9700 1 -- 172.21.15.190:0/3511528319 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8fc014420 con 0x7fb90c0f62a0 2023-11-15T04:30:34.842 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.842+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb90c107ed0 con 0x7fb90c0f62a0 2023-11-15T04:30:34.843 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.843+0000 7fb908ff9700 1 -- 172.21.15.190:0/3511528319 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8fc010230 con 0x7fb90c0f62a0 2023-11-15T04:30:34.843 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:34.843 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:34.845 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8f402ff60 msgr2=0x7fb8f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:34.845 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 --2- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8f402ff60 0x7fb8f4032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb9000096a0 tx=0x7fb900006b40).stop 2023-11-15T04:30:34.845 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 msgr2=0x7fb90c1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:34.845 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 --2- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 0x7fb90c1064a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fb8fc013510 tx=0x7fb8fc00b040).stop 2023-11-15T04:30:34.846 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 shutdown_connections 2023-11-15T04:30:34.846 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 --2- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8f402ff60 0x7fb8f4032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:34.846 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 --2- 172.21.15.190:0/3511528319 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90c0f62a0 0x7fb90c1064a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:34.846 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 >> 172.21.15.190:0/3511528319 conn(0x7fb90c0f1ac0 msgr2=0x7fb90c0f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:34.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 shutdown_connections 2023-11-15T04:30:34.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:34.845+0000 7fb911579700 1 -- 172.21.15.190:0/3511528319 wait complete. 2023-11-15T04:30:34.849 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:35.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:35 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:34.843541+0000 mon.smithi047 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.190:0/3511528319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:36.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:36.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:35.067743+0000 mgr.smithi047.zgysuz (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:37.336 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.335+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2424075318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 msgr2=0x7f7a180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:37.337 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.335+0000 7f7a1e2fd700 1 --2- 172.21.15.190:0/2424075318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 0x7f7a180f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f7a08009fb0 tx=0x7f7a08009b30).stop 2023-11-15T04:30:37.337 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.335+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2424075318 shutdown_connections 2023-11-15T04:30:37.337 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.335+0000 7f7a1e2fd700 1 --2- 172.21.15.190:0/2424075318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 0x7f7a180f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:37.337 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.335+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2424075318 >> 172.21.15.190:0/2424075318 conn(0x7f7a180f0b70 msgr2=0x7f7a180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:37.338 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.335+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2424075318 shutdown_connections 2023-11-15T04:30:37.338 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.335+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2424075318 wait complete. 2023-11-15T04:30:37.338 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.335+0000 7f7a1e2fd700 1 Processor -- start 2023-11-15T04:30:37.338 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.336+0000 7f7a1e2fd700 1 -- start start 2023-11-15T04:30:37.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.336+0000 7f7a1e2fd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 0x7f7a18105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:37.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.336+0000 7f7a1e2fd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18105b20 con 0x7f7a180f5410 2023-11-15T04:30:37.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.336+0000 7f7a17fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 0x7f7a18105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:37.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.336+0000 7f7a17fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 0x7f7a18105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34796/0 (socket says 172.21.15.190:34796) 2023-11-15T04:30:37.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.336+0000 7f7a17fff700 1 -- 172.21.15.190:0/2418849158 learned_addr learned my addr 172.21.15.190:0/2418849158 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:37.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.336+0000 7f7a17fff700 1 -- 172.21.15.190:0/2418849158 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a08005040 con 0x7f7a180f5410 2023-11-15T04:30:37.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.336+0000 7f7a17fff700 1 --2- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 0x7f7a18105630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f7a08013510 tx=0x7f7a0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:37.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.337+0000 7f7a14ff9700 1 -- 172.21.15.190:0/2418849158 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a08009320 con 0x7f7a180f5410 2023-11-15T04:30:37.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.337+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a18105d20 con 0x7f7a180f5410 2023-11-15T04:30:37.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.337+0000 7f7a14ff9700 1 -- 172.21.15.190:0/2418849158 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a08013f00 con 0x7f7a180f5410 2023-11-15T04:30:37.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.337+0000 7f7a14ff9700 1 -- 172.21.15.190:0/2418849158 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a08014d90 con 0x7f7a180f5410 2023-11-15T04:30:37.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.337+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a181061b0 con 0x7f7a180f5410 2023-11-15T04:30:37.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.338+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a18045ad0 con 0x7f7a180f5410 2023-11-15T04:30:37.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.338+0000 7f7a14ff9700 1 -- 172.21.15.190:0/2418849158 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a080095f0 con 0x7f7a180f5410 2023-11-15T04:30:37.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.338+0000 7f7a14ff9700 1 --2- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a0402ff60 0x7f7a04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:37.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.338+0000 7f7a14ff9700 1 -- 172.21.15.190:0/2418849158 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a08040fc0 con 0x7f7a180f5410 2023-11-15T04:30:37.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.338+0000 7f7a177fe700 1 --2- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a0402ff60 0x7f7a04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:37.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.338+0000 7f7a177fe700 1 --2- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a0402ff60 0x7f7a04032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7a000096a0 tx=0x7f7a00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:37.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.343+0000 7f7a14ff9700 1 -- 172.21.15.190:0/2418849158 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a08014420 con 0x7f7a180f5410 2023-11-15T04:30:37.550 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.550+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a180fae60 con 0x7f7a180f5410 2023-11-15T04:30:37.550 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.550+0000 7f7a14ff9700 1 -- 172.21.15.190:0/2418849158 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a08010230 con 0x7f7a180f5410 2023-11-15T04:30:37.551 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:37.551 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:37.552 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a0402ff60 msgr2=0x7f7a04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:37.552 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 --2- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a0402ff60 0x7f7a04032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7a000096a0 tx=0x7f7a00006b40).stop 2023-11-15T04:30:37.553 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 msgr2=0x7f7a18105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:37.553 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 --2- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 0x7f7a18105630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f7a08013510 tx=0x7f7a0800b040).stop 2023-11-15T04:30:37.553 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 shutdown_connections 2023-11-15T04:30:37.553 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 --2- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a0402ff60 0x7f7a04032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:37.554 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 --2- 172.21.15.190:0/2418849158 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a180f5410 0x7f7a18105630 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:37.554 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 >> 172.21.15.190:0/2418849158 conn(0x7f7a180f0b70 msgr2=0x7f7a180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:37.554 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 shutdown_connections 2023-11-15T04:30:37.554 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:37.552+0000 7f7a1e2fd700 1 -- 172.21.15.190:0/2418849158 wait complete. 2023-11-15T04:30:37.555 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:38.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:37.068091+0000 mgr.smithi047.zgysuz (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:38.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:37.550842+0000 mon.smithi047 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.190:0/2418849158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:39.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:39.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:40.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.339+0000 7f38d6d4b700 1 -- 172.21.15.190:0/3631787985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 msgr2=0x7f38d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:40.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.339+0000 7f38d6d4b700 1 --2- 172.21.15.190:0/3631787985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 0x7f38d00f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f38c4009fb0 tx=0x7f38c4009b30).stop 2023-11-15T04:30:40.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.339+0000 7f38d6d4b700 1 -- 172.21.15.190:0/3631787985 shutdown_connections 2023-11-15T04:30:40.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.339+0000 7f38d6d4b700 1 --2- 172.21.15.190:0/3631787985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 0x7f38d00f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:40.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.339+0000 7f38d6d4b700 1 -- 172.21.15.190:0/3631787985 >> 172.21.15.190:0/3631787985 conn(0x7f38d00f1c70 msgr2=0x7f38d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:40.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.339+0000 7f38d6d4b700 1 -- 172.21.15.190:0/3631787985 shutdown_connections 2023-11-15T04:30:40.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.339+0000 7f38d6d4b700 1 -- 172.21.15.190:0/3631787985 wait complete. 2023-11-15T04:30:40.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.340+0000 7f38d6d4b700 1 Processor -- start 2023-11-15T04:30:40.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.340+0000 7f38d6d4b700 1 -- start start 2023-11-15T04:30:40.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.340+0000 7f38d6d4b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 0x7f38d006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:40.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.340+0000 7f38d6d4b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d0069e20 con 0x7f38d00f6510 2023-11-15T04:30:40.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.340+0000 7f38d4ae7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 0x7f38d006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:40.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.340+0000 7f38d4ae7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 0x7f38d006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:52778/0 (socket says 172.21.15.190:52778) 2023-11-15T04:30:40.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.340+0000 7f38d4ae7700 1 -- 172.21.15.190:0/2226633364 learned_addr learned my addr 172.21.15.190:0/2226633364 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:40.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.341+0000 7f38d4ae7700 1 -- 172.21.15.190:0/2226633364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c4005040 con 0x7f38d00f6510 2023-11-15T04:30:40.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.341+0000 7f38d4ae7700 1 --2- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 0x7f38d006b9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f38c4013510 tx=0x7f38c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:40.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.341+0000 7f38cd7fa700 1 -- 172.21.15.190:0/2226633364 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c4009320 con 0x7f38d00f6510 2023-11-15T04:30:40.345 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.341+0000 7f38cd7fa700 1 -- 172.21.15.190:0/2226633364 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38c4013f00 con 0x7f38d00f6510 2023-11-15T04:30:40.345 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.341+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d0069fc0 con 0x7f38d00f6510 2023-11-15T04:30:40.345 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.341+0000 7f38cd7fa700 1 -- 172.21.15.190:0/2226633364 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c4014d90 con 0x7f38d00f6510 2023-11-15T04:30:40.345 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.341+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d006a450 con 0x7f38d00f6510 2023-11-15T04:30:40.346 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.342+0000 7f38cd7fa700 1 -- 172.21.15.190:0/2226633364 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38c40095f0 con 0x7f38d00f6510 2023-11-15T04:30:40.347 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.342+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d0059100 con 0x7f38d00f6510 2023-11-15T04:30:40.347 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.342+0000 7f38cd7fa700 1 --2- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38c002ff60 0x7f38c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:40.347 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.342+0000 7f38cd7fa700 1 -- 172.21.15.190:0/2226633364 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38c4041080 con 0x7f38d00f6510 2023-11-15T04:30:40.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.342+0000 7f38cffff700 1 --2- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38c002ff60 0x7f38c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:40.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.343+0000 7f38cffff700 1 --2- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38c002ff60 0x7f38c0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f38bc0096a0 tx=0x7f38bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:40.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.347+0000 7f38cd7fa700 1 -- 172.21.15.190:0/2226633364 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38c4014420 con 0x7f38d00f6510 2023-11-15T04:30:40.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:39.068417+0000 mgr.smithi047.zgysuz (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:40.554 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.554+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38d00fbf60 con 0x7f38d00f6510 2023-11-15T04:30:40.555 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.555+0000 7f38cd7fa700 1 -- 172.21.15.190:0/2226633364 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38c401d7f0 con 0x7f38d00f6510 2023-11-15T04:30:40.555 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:40.556 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:40.556 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.556+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38c002ff60 msgr2=0x7f38c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:40.557 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.556+0000 7f38d6d4b700 1 --2- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38c002ff60 0x7f38c0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f38bc0096a0 tx=0x7f38bc006b40).stop 2023-11-15T04:30:40.557 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.556+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 msgr2=0x7f38d006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:40.557 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.556+0000 7f38d6d4b700 1 --2- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 0x7f38d006b9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f38c4013510 tx=0x7f38c400b040).stop 2023-11-15T04:30:40.557 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.556+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 shutdown_connections 2023-11-15T04:30:40.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.556+0000 7f38d6d4b700 1 --2- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38c002ff60 0x7f38c0032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:40.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.556+0000 7f38d6d4b700 1 --2- 172.21.15.190:0/2226633364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38d00f6510 0x7f38d006b9a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:40.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.556+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 >> 172.21.15.190:0/2226633364 conn(0x7f38d00f1c70 msgr2=0x7f38d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:40.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.556+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 shutdown_connections 2023-11-15T04:30:40.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:40.557+0000 7f38d6d4b700 1 -- 172.21.15.190:0/2226633364 wait complete. 2023-11-15T04:30:40.559 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:41.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:41 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:40.555319+0000 mon.smithi047 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.190:0/2226633364' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:41.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:41.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:42.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:41.068722+0000 mgr.smithi047.zgysuz (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:43.080 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.079+0000 7f1aab96e700 1 -- 172.21.15.190:0/2283936264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 msgr2=0x7f1aa40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:43.081 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.079+0000 7f1aab96e700 1 --2- 172.21.15.190:0/2283936264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 0x7f1aa40f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f1a94009fb0 tx=0x7f1a94009b30).stop 2023-11-15T04:30:43.081 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.079+0000 7f1aab96e700 1 -- 172.21.15.190:0/2283936264 shutdown_connections 2023-11-15T04:30:43.081 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.079+0000 7f1aab96e700 1 --2- 172.21.15.190:0/2283936264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 0x7f1aa40f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:43.081 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.079+0000 7f1aab96e700 1 -- 172.21.15.190:0/2283936264 >> 172.21.15.190:0/2283936264 conn(0x7f1aa40f0b70 msgr2=0x7f1aa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:43.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.079+0000 7f1aab96e700 1 -- 172.21.15.190:0/2283936264 shutdown_connections 2023-11-15T04:30:43.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.079+0000 7f1aab96e700 1 -- 172.21.15.190:0/2283936264 wait complete. 2023-11-15T04:30:43.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.080+0000 7f1aab96e700 1 Processor -- start 2023-11-15T04:30:43.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.080+0000 7f1aab96e700 1 -- start start 2023-11-15T04:30:43.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.080+0000 7f1aab96e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 0x7f1aa4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:43.083 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.080+0000 7f1aab96e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa4060240 con 0x7f1aa40f5410 2023-11-15T04:30:43.083 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.080+0000 7f1aa970a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 0x7f1aa4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:43.083 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.080+0000 7f1aa970a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 0x7f1aa4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:52808/0 (socket says 172.21.15.190:52808) 2023-11-15T04:30:43.083 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.080+0000 7f1aa970a700 1 -- 172.21.15.190:0/2360102863 learned_addr learned my addr 172.21.15.190:0/2360102863 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:43.083 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.080+0000 7f1aa970a700 1 -- 172.21.15.190:0/2360102863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a94005040 con 0x7f1aa40f5410 2023-11-15T04:30:43.084 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.080+0000 7f1aa970a700 1 --2- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 0x7f1aa4109430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1a94013510 tx=0x7f1a9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:43.084 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.081+0000 7f1a99ffb700 1 -- 172.21.15.190:0/2360102863 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a94009320 con 0x7f1aa40f5410 2023-11-15T04:30:43.084 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.081+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa4060440 con 0x7f1aa40f5410 2023-11-15T04:30:43.084 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.081+0000 7f1a99ffb700 1 -- 172.21.15.190:0/2360102863 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a94013f00 con 0x7f1aa40f5410 2023-11-15T04:30:43.085 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.081+0000 7f1a99ffb700 1 -- 172.21.15.190:0/2360102863 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a94014d90 con 0x7f1aa40f5410 2023-11-15T04:30:43.085 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.081+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa4109ae0 con 0x7f1aa40f5410 2023-11-15T04:30:43.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.082+0000 7f1a99ffb700 1 -- 172.21.15.190:0/2360102863 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a940095f0 con 0x7f1aa40f5410 2023-11-15T04:30:43.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.082+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa4045ad0 con 0x7f1aa40f5410 2023-11-15T04:30:43.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.082+0000 7f1a99ffb700 1 --2- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1a9002ff60 0x7f1a90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:43.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.082+0000 7f1a99ffb700 1 -- 172.21.15.190:0/2360102863 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a940410c0 con 0x7f1aa40f5410 2023-11-15T04:30:43.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.082+0000 7f1aa8f09700 1 --2- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1a9002ff60 0x7f1a90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:43.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.083+0000 7f1aa8f09700 1 --2- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1a9002ff60 0x7f1a90032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1aa00096a0 tx=0x7f1aa0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:43.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.087+0000 7f1a99ffb700 1 -- 172.21.15.190:0/2360102863 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a9401d7f0 con 0x7f1aa40f5410 2023-11-15T04:30:43.290 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.291+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1aa40fae60 con 0x7f1aa40f5410 2023-11-15T04:30:43.291 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.291+0000 7f1a99ffb700 1 -- 172.21.15.190:0/2360102863 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a94010230 con 0x7f1aa40f5410 2023-11-15T04:30:43.291 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:43.292 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:43.293 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1a9002ff60 msgr2=0x7f1a90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:43.294 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 --2- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1a9002ff60 0x7f1a90032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1aa00096a0 tx=0x7f1aa0006b40).stop 2023-11-15T04:30:43.294 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 msgr2=0x7f1aa4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:43.294 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 --2- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 0x7f1aa4109430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1a94013510 tx=0x7f1a9400b040).stop 2023-11-15T04:30:43.294 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 shutdown_connections 2023-11-15T04:30:43.295 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 --2- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1a9002ff60 0x7f1a90032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:43.295 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 --2- 172.21.15.190:0/2360102863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1aa40f5410 0x7f1aa4109430 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:43.296 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 >> 172.21.15.190:0/2360102863 conn(0x7f1aa40f0b70 msgr2=0x7f1aa40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:43.296 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 shutdown_connections 2023-11-15T04:30:43.296 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:43.293+0000 7f1aab96e700 1 -- 172.21.15.190:0/2360102863 wait complete. 2023-11-15T04:30:43.297 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:43.069002+0000 mgr.smithi047.zgysuz (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:43.291893+0000 mon.smithi047 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.190:0/2360102863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:44.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:44.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:45.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.771+0000 7f06f3391700 1 -- 172.21.15.190:0/2605484111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 msgr2=0x7f06ec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:45.773 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.771+0000 7f06f3391700 1 --2- 172.21.15.190:0/2605484111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 0x7f06ec0f66c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f06dc009fb0 tx=0x7f06dc009b30).stop 2023-11-15T04:30:45.773 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.771+0000 7f06f3391700 1 -- 172.21.15.190:0/2605484111 shutdown_connections 2023-11-15T04:30:45.773 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.771+0000 7f06f3391700 1 --2- 172.21.15.190:0/2605484111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 0x7f06ec0f66c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:45.773 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.771+0000 7f06f3391700 1 -- 172.21.15.190:0/2605484111 >> 172.21.15.190:0/2605484111 conn(0x7f06ec0f1ac0 msgr2=0x7f06ec0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:45.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.771+0000 7f06f3391700 1 -- 172.21.15.190:0/2605484111 shutdown_connections 2023-11-15T04:30:45.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.771+0000 7f06f3391700 1 -- 172.21.15.190:0/2605484111 wait complete. 2023-11-15T04:30:45.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.772+0000 7f06f3391700 1 Processor -- start 2023-11-15T04:30:45.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.772+0000 7f06f3391700 1 -- start start 2023-11-15T04:30:45.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.772+0000 7f06f3391700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 0x7f06ec1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:45.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.772+0000 7f06f3391700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec106990 con 0x7f06ec0f62a0 2023-11-15T04:30:45.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.772+0000 7f06f238f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 0x7f06ec1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:45.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.772+0000 7f06f238f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 0x7f06ec1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:52820/0 (socket says 172.21.15.190:52820) 2023-11-15T04:30:45.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.772+0000 7f06f238f700 1 -- 172.21.15.190:0/3505901890 learned_addr learned my addr 172.21.15.190:0/3505901890 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:45.776 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.772+0000 7f06f238f700 1 -- 172.21.15.190:0/3505901890 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc005040 con 0x7f06ec0f62a0 2023-11-15T04:30:45.776 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.773+0000 7f06f238f700 1 --2- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 0x7f06ec1064a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f06dc013510 tx=0x7f06dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:45.776 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.773+0000 7f06e2ffd700 1 -- 172.21.15.190:0/3505901890 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06dc009320 con 0x7f06ec0f62a0 2023-11-15T04:30:45.776 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.773+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec106b90 con 0x7f06ec0f62a0 2023-11-15T04:30:45.777 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.773+0000 7f06e2ffd700 1 -- 172.21.15.190:0/3505901890 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06dc013f00 con 0x7f06ec0f62a0 2023-11-15T04:30:45.777 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.773+0000 7f06e2ffd700 1 -- 172.21.15.190:0/3505901890 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06dc014dc0 con 0x7f06ec0f62a0 2023-11-15T04:30:45.777 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.773+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec107020 con 0x7f06ec0f62a0 2023-11-15T04:30:45.778 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.773+0000 7f06e2ffd700 1 -- 172.21.15.190:0/3505901890 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06dc0095f0 con 0x7f06ec0f62a0 2023-11-15T04:30:45.778 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.774+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec045ad0 con 0x7f06ec0f62a0 2023-11-15T04:30:45.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.774+0000 7f06e2ffd700 1 --2- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f06d802ff60 0x7f06d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:45.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.774+0000 7f06e2ffd700 1 -- 172.21.15.190:0/3505901890 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06dc041040 con 0x7f06ec0f62a0 2023-11-15T04:30:45.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.774+0000 7f06f1b8e700 1 --2- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f06d802ff60 0x7f06d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:45.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.774+0000 7f06f1b8e700 1 --2- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f06d802ff60 0x7f06d8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f06e80096a0 tx=0x7f06e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:45.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.779+0000 7f06e2ffd700 1 -- 172.21.15.190:0/3505901890 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06dc014420 con 0x7f06ec0f62a0 2023-11-15T04:30:45.983 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.983+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06ec107ed0 con 0x7f06ec0f62a0 2023-11-15T04:30:45.984 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.984+0000 7f06e2ffd700 1 -- 172.21.15.190:0/3505901890 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f06dc010230 con 0x7f06ec0f62a0 2023-11-15T04:30:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:45.985 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:45.986 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f06d802ff60 msgr2=0x7f06d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:45.986 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 --2- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f06d802ff60 0x7f06d8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f06e80096a0 tx=0x7f06e8006b40).stop 2023-11-15T04:30:45.987 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 msgr2=0x7f06ec1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:45.987 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 --2- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 0x7f06ec1064a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f06dc013510 tx=0x7f06dc00b040).stop 2023-11-15T04:30:45.987 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 shutdown_connections 2023-11-15T04:30:45.987 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 --2- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f06d802ff60 0x7f06d8032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:45.988 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 --2- 172.21.15.190:0/3505901890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06ec0f62a0 0x7f06ec1064a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:45.988 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 >> 172.21.15.190:0/3505901890 conn(0x7f06ec0f1ac0 msgr2=0x7f06ec0f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:45.988 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 shutdown_connections 2023-11-15T04:30:45.988 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:45.986+0000 7f06f3391700 1 -- 172.21.15.190:0/3505901890 wait complete. 2023-11-15T04:30:45.989 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:45.069353+0000 mgr.smithi047.zgysuz (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:45.984724+0000 mon.smithi047 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.190:0/3505901890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:47.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:47.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:47.069682+0000 mgr.smithi047.zgysuz (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:48.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.419+0000 7f978ade2700 1 -- 172.21.15.190:0/4019730293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 msgr2=0x7f97840f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:48.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.419+0000 7f978ade2700 1 --2- 172.21.15.190:0/4019730293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 0x7f97840f4be0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f9778009fb0 tx=0x7f9778009b30).stop 2023-11-15T04:30:48.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.419+0000 7f978ade2700 1 -- 172.21.15.190:0/4019730293 shutdown_connections 2023-11-15T04:30:48.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.419+0000 7f978ade2700 1 --2- 172.21.15.190:0/4019730293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 0x7f97840f4be0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:48.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.419+0000 7f978ade2700 1 -- 172.21.15.190:0/4019730293 >> 172.21.15.190:0/4019730293 conn(0x7f97840f02e0 msgr2=0x7f97840f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:48.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.419+0000 7f978ade2700 1 -- 172.21.15.190:0/4019730293 shutdown_connections 2023-11-15T04:30:48.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.419+0000 7f978ade2700 1 -- 172.21.15.190:0/4019730293 wait complete. 2023-11-15T04:30:48.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.419+0000 7f978ade2700 1 Processor -- start 2023-11-15T04:30:48.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.419+0000 7f978ade2700 1 -- start start 2023-11-15T04:30:48.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.420+0000 7f978ade2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 0x7f97840f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:48.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.420+0000 7f978ade2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97840f57f0 con 0x7f97840f47c0 2023-11-15T04:30:48.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.420+0000 7f9788b7e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 0x7f97840f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:48.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.420+0000 7f9788b7e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 0x7f97840f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:52836/0 (socket says 172.21.15.190:52836) 2023-11-15T04:30:48.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.420+0000 7f9788b7e700 1 -- 172.21.15.190:0/2825863216 learned_addr learned my addr 172.21.15.190:0/2825863216 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:48.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.420+0000 7f9788b7e700 1 -- 172.21.15.190:0/2825863216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778005040 con 0x7f97840f47c0 2023-11-15T04:30:48.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.420+0000 7f9788b7e700 1 --2- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 0x7f97840f7370 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9778013510 tx=0x7f977800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:48.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.421+0000 7f97817fa700 1 -- 172.21.15.190:0/2825863216 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9778009320 con 0x7f97840f47c0 2023-11-15T04:30:48.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.421+0000 7f97817fa700 1 -- 172.21.15.190:0/2825863216 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9778013f00 con 0x7f97840f47c0 2023-11-15T04:30:48.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.421+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97840f5990 con 0x7f97840f47c0 2023-11-15T04:30:48.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.421+0000 7f97817fa700 1 -- 172.21.15.190:0/2825863216 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9778014d90 con 0x7f97840f47c0 2023-11-15T04:30:48.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.421+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97840f5e20 con 0x7f97840f47c0 2023-11-15T04:30:48.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.421+0000 7f97817fa700 1 -- 172.21.15.190:0/2825863216 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97780095f0 con 0x7f97840f47c0 2023-11-15T04:30:48.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.422+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9784059100 con 0x7f97840f47c0 2023-11-15T04:30:48.426 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.422+0000 7f97817fa700 1 --2- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f977402ff60 0x7f9774032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:48.426 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.422+0000 7f97817fa700 1 -- 172.21.15.190:0/2825863216 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9778041000 con 0x7f97840f47c0 2023-11-15T04:30:48.426 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.422+0000 7f9783fff700 1 --2- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f977402ff60 0x7f9774032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:48.426 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.422+0000 7f9783fff700 1 --2- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f977402ff60 0x7f9774032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f97700096a0 tx=0x7f9770006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:48.427 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.427+0000 7f97817fa700 1 -- 172.21.15.190:0/2825863216 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9778014420 con 0x7f97840f47c0 2023-11-15T04:30:48.634 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.634+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97840fbf60 con 0x7f97840f47c0 2023-11-15T04:30:48.634 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.634+0000 7f97817fa700 1 -- 172.21.15.190:0/2825863216 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f977801d7f0 con 0x7f97840f47c0 2023-11-15T04:30:48.635 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:48.635 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:48.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f977402ff60 msgr2=0x7f9774032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:48.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 --2- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f977402ff60 0x7f9774032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f97700096a0 tx=0x7f9770006b40).stop 2023-11-15T04:30:48.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 msgr2=0x7f97840f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:48.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 --2- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 0x7f97840f7370 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9778013510 tx=0x7f977800b040).stop 2023-11-15T04:30:48.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 shutdown_connections 2023-11-15T04:30:48.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 --2- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f977402ff60 0x7f9774032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:48.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 --2- 172.21.15.190:0/2825863216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97840f47c0 0x7f97840f7370 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:48.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 >> 172.21.15.190:0/2825863216 conn(0x7f97840f02e0 msgr2=0x7f97840f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:48.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 shutdown_connections 2023-11-15T04:30:48.640 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:48.636+0000 7f978ade2700 1 -- 172.21.15.190:0/2825863216 wait complete. 2023-11-15T04:30:48.640 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:49.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:48 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:48.634939+0000 mon.smithi047 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.190:0/2825863216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:50.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:49 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:49.069998+0000 mgr.smithi047.zgysuz (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:50.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:50.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:51.184 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 -- 172.21.15.190:0/1409783041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 msgr2=0x7f79340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:51.185 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 --2- 172.21.15.190:0/1409783041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 0x7f79340f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f7930009fb0 tx=0x7f7930009b30).stop 2023-11-15T04:30:51.185 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 -- 172.21.15.190:0/1409783041 shutdown_connections 2023-11-15T04:30:51.185 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 --2- 172.21.15.190:0/1409783041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 0x7f79340f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:51.185 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 -- 172.21.15.190:0/1409783041 >> 172.21.15.190:0/1409783041 conn(0x7f79340f0b70 msgr2=0x7f79340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:51.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 -- 172.21.15.190:0/1409783041 shutdown_connections 2023-11-15T04:30:51.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 -- 172.21.15.190:0/1409783041 wait complete. 2023-11-15T04:30:51.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 Processor -- start 2023-11-15T04:30:51.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 -- start start 2023-11-15T04:30:51.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 0x7f793410b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:51.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.183+0000 7f793c628700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7934107210 con 0x7f79340f5410 2023-11-15T04:30:51.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.184+0000 7f793a3c4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 0x7f793410b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:51.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.184+0000 7f793a3c4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 0x7f793410b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:43956/0 (socket says 172.21.15.190:43956) 2023-11-15T04:30:51.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.184+0000 7f793a3c4700 1 -- 172.21.15.190:0/1667358967 learned_addr learned my addr 172.21.15.190:0/1667358967 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:51.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.184+0000 7f793a3c4700 1 -- 172.21.15.190:0/1667358967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7930005040 con 0x7f79340f5410 2023-11-15T04:30:51.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.184+0000 7f793a3c4700 1 --2- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 0x7f793410b820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f7930014000 tx=0x7f793000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:51.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.184+0000 7f792affd700 1 -- 172.21.15.190:0/1667358967 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79300095f0 con 0x7f79340f5410 2023-11-15T04:30:51.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.184+0000 7f792affd700 1 -- 172.21.15.190:0/1667358967 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7930007e70 con 0x7f79340f5410 2023-11-15T04:30:51.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.184+0000 7f792affd700 1 -- 172.21.15.190:0/1667358967 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7930014ea0 con 0x7f79340f5410 2023-11-15T04:30:51.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.184+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7934060220 con 0x7f79340f5410 2023-11-15T04:30:51.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.185+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793410bd10 con 0x7f79340f5410 2023-11-15T04:30:51.191 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.185+0000 7f792affd700 1 -- 172.21.15.190:0/1667358967 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7930025070 con 0x7f79340f5410 2023-11-15T04:30:51.191 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.185+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7934045ad0 con 0x7f79340f5410 2023-11-15T04:30:51.191 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.186+0000 7f792affd700 1 --2- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f792002ff60 0x7f7920032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:51.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.186+0000 7f792affd700 1 -- 172.21.15.190:0/1667358967 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7930040a10 con 0x7f79340f5410 2023-11-15T04:30:51.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.186+0000 7f7939bc3700 1 --2- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f792002ff60 0x7f7920032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:51.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.186+0000 7f7939bc3700 1 --2- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f792002ff60 0x7f7920032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f79240096a0 tx=0x7f7924006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:51.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.190+0000 7f792affd700 1 -- 172.21.15.190:0/1667358967 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79300079c0 con 0x7f79340f5410 2023-11-15T04:30:51.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.395+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79340fae60 con 0x7f79340f5410 2023-11-15T04:30:51.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.396+0000 7f792affd700 1 -- 172.21.15.190:0/1667358967 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f793000f300 con 0x7f79340f5410 2023-11-15T04:30:51.396 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:51.396 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:51.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.397+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f792002ff60 msgr2=0x7f7920032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:51.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.397+0000 7f793c628700 1 --2- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f792002ff60 0x7f7920032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f79240096a0 tx=0x7f7924006b40).stop 2023-11-15T04:30:51.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.397+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 msgr2=0x7f793410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:51.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.397+0000 7f793c628700 1 --2- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 0x7f793410b820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f7930014000 tx=0x7f793000b040).stop 2023-11-15T04:30:51.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.397+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 shutdown_connections 2023-11-15T04:30:51.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.397+0000 7f793c628700 1 --2- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f792002ff60 0x7f7920032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:51.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.397+0000 7f793c628700 1 --2- 172.21.15.190:0/1667358967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79340f5410 0x7f793410b820 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:51.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.397+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 >> 172.21.15.190:0/1667358967 conn(0x7f79340f0b70 msgr2=0x7f79340f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:51.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.398+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 shutdown_connections 2023-11-15T04:30:51.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:51.398+0000 7f793c628700 1 -- 172.21.15.190:0/1667358967 wait complete. 2023-11-15T04:30:51.400 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:51.070341+0000 mgr.smithi047.zgysuz (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:51.396114+0000 mon.smithi047 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.190:0/1667358967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:52.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:52.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:53.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.882+0000 7fcaf09be700 1 -- 172.21.15.190:0/950848505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 msgr2=0x7fcaec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:53.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.882+0000 7fcaf09be700 1 --2- 172.21.15.190:0/950848505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 0x7fcaec0f55c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fcadc009fb0 tx=0x7fcadc009b30).stop 2023-11-15T04:30:53.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.882+0000 7fcaf09be700 1 -- 172.21.15.190:0/950848505 shutdown_connections 2023-11-15T04:30:53.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.882+0000 7fcaf09be700 1 --2- 172.21.15.190:0/950848505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 0x7fcaec0f55c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:53.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.882+0000 7fcaf09be700 1 -- 172.21.15.190:0/950848505 >> 172.21.15.190:0/950848505 conn(0x7fcaec0f09a0 msgr2=0x7fcaec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:53.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.882+0000 7fcaf09be700 1 -- 172.21.15.190:0/950848505 shutdown_connections 2023-11-15T04:30:53.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.882+0000 7fcaf09be700 1 -- 172.21.15.190:0/950848505 wait complete. 2023-11-15T04:30:53.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.882+0000 7fcaf09be700 1 Processor -- start 2023-11-15T04:30:53.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.882+0000 7fcaf09be700 1 -- start start 2023-11-15T04:30:53.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.883+0000 7fcaf09be700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 0x7fcaec1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:53.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.883+0000 7fcaf09be700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec1058a0 con 0x7fcaec0f51a0 2023-11-15T04:30:53.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.883+0000 7fcaeb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 0x7fcaec1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:53.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.883+0000 7fcaeb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 0x7fcaec1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:43982/0 (socket says 172.21.15.190:43982) 2023-11-15T04:30:53.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.883+0000 7fcaeb7fe700 1 -- 172.21.15.190:0/3677296664 learned_addr learned my addr 172.21.15.190:0/3677296664 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:53.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.883+0000 7fcaeb7fe700 1 -- 172.21.15.190:0/3677296664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcadc005040 con 0x7fcaec0f51a0 2023-11-15T04:30:53.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.884+0000 7fcaeb7fe700 1 --2- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 0x7fcaec1053b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcadc013510 tx=0x7fcadc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:53.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.884+0000 7fcad3fff700 1 -- 172.21.15.190:0/3677296664 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcadc009320 con 0x7fcaec0f51a0 2023-11-15T04:30:53.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.884+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaec105aa0 con 0x7fcaec0f51a0 2023-11-15T04:30:53.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.884+0000 7fcad3fff700 1 -- 172.21.15.190:0/3677296664 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcadc013f00 con 0x7fcaec0f51a0 2023-11-15T04:30:53.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.884+0000 7fcad3fff700 1 -- 172.21.15.190:0/3677296664 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcadc014dc0 con 0x7fcaec0f51a0 2023-11-15T04:30:53.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.884+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaec105f30 con 0x7fcaec0f51a0 2023-11-15T04:30:53.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.885+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaec045ad0 con 0x7fcaec0f51a0 2023-11-15T04:30:53.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.885+0000 7fcad3fff700 1 -- 172.21.15.190:0/3677296664 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcadc0095f0 con 0x7fcaec0f51a0 2023-11-15T04:30:53.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.885+0000 7fcad3fff700 1 --2- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcad402ff60 0x7fcad4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:53.890 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.885+0000 7fcad3fff700 1 -- 172.21.15.190:0/3677296664 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcadc041000 con 0x7fcaec0f51a0 2023-11-15T04:30:53.891 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.885+0000 7fcaeaffd700 1 --2- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcad402ff60 0x7fcad4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:53.893 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.885+0000 7fcaeaffd700 1 --2- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcad402ff60 0x7fcad4032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fcae00096a0 tx=0x7fcae0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:53.894 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:53.890+0000 7fcad3fff700 1 -- 172.21.15.190:0/3677296664 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcadc01c790 con 0x7fcaec0f51a0 2023-11-15T04:30:54.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.100+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcaec106de0 con 0x7fcaec0f51a0 2023-11-15T04:30:54.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.101+0000 7fcad3fff700 1 -- 172.21.15.190:0/3677296664 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcadc010230 con 0x7fcaec0f51a0 2023-11-15T04:30:54.102 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:54.102 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:54.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.102+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcad402ff60 msgr2=0x7fcad4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:54.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.102+0000 7fcaf09be700 1 --2- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcad402ff60 0x7fcad4032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fcae00096a0 tx=0x7fcae0006b40).stop 2023-11-15T04:30:54.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.102+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 msgr2=0x7fcaec1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:54.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.102+0000 7fcaf09be700 1 --2- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 0x7fcaec1053b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcadc013510 tx=0x7fcadc00b040).stop 2023-11-15T04:30:54.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.102+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 shutdown_connections 2023-11-15T04:30:54.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.102+0000 7fcaf09be700 1 --2- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcad402ff60 0x7fcad4032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:54.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.102+0000 7fcaf09be700 1 --2- 172.21.15.190:0/3677296664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaec0f51a0 0x7fcaec1053b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:54.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.102+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 >> 172.21.15.190:0/3677296664 conn(0x7fcaec0f09a0 msgr2=0x7fcaec0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:54.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.103+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 shutdown_connections 2023-11-15T04:30:54.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:54.103+0000 7fcaf09be700 1 -- 172.21.15.190:0/3677296664 wait complete. 2023-11-15T04:30:54.105 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:53.070683+0000 mgr.smithi047.zgysuz (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:54.101349+0000 mon.smithi047 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.190:0/3677296664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:55.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:55.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:55.071071+0000 mgr.smithi047.zgysuz (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:56.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.603+0000 7f89c84ae700 1 -- 172.21.15.190:0/4261201525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 msgr2=0x7f89c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:56.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.603+0000 7f89c84ae700 1 --2- 172.21.15.190:0/4261201525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 0x7f89c00f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f89b8009fb0 tx=0x7f89b8009b30).stop 2023-11-15T04:30:56.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.603+0000 7f89c84ae700 1 -- 172.21.15.190:0/4261201525 shutdown_connections 2023-11-15T04:30:56.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.603+0000 7f89c84ae700 1 --2- 172.21.15.190:0/4261201525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 0x7f89c00f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:56.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.603+0000 7f89c84ae700 1 -- 172.21.15.190:0/4261201525 >> 172.21.15.190:0/4261201525 conn(0x7f89c00f0b70 msgr2=0x7f89c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:56.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.603+0000 7f89c84ae700 1 -- 172.21.15.190:0/4261201525 shutdown_connections 2023-11-15T04:30:56.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.603+0000 7f89c84ae700 1 -- 172.21.15.190:0/4261201525 wait complete. 2023-11-15T04:30:56.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.603+0000 7f89c84ae700 1 Processor -- start 2023-11-15T04:30:56.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.604+0000 7f89c84ae700 1 -- start start 2023-11-15T04:30:56.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.604+0000 7f89c84ae700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 0x7f89c0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:56.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.604+0000 7f89c84ae700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c0060220 con 0x7f89c00f5410 2023-11-15T04:30:56.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.604+0000 7f89c624a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 0x7f89c0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:56.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.604+0000 7f89c624a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 0x7f89c0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:43994/0 (socket says 172.21.15.190:43994) 2023-11-15T04:30:56.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.604+0000 7f89c624a700 1 -- 172.21.15.190:0/409631136 learned_addr learned my addr 172.21.15.190:0/409631136 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:56.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.604+0000 7f89c624a700 1 -- 172.21.15.190:0/409631136 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b8005040 con 0x7f89c00f5410 2023-11-15T04:30:56.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.604+0000 7f89c624a700 1 --2- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 0x7f89c0109450 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f89b8013510 tx=0x7f89b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:56.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.605+0000 7f89beffd700 1 -- 172.21.15.190:0/409631136 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b8009320 con 0x7f89c00f5410 2023-11-15T04:30:56.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.605+0000 7f89beffd700 1 -- 172.21.15.190:0/409631136 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89b8013f00 con 0x7f89c00f5410 2023-11-15T04:30:56.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.605+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c0060420 con 0x7f89c00f5410 2023-11-15T04:30:56.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.605+0000 7f89beffd700 1 -- 172.21.15.190:0/409631136 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b8014d90 con 0x7f89c00f5410 2023-11-15T04:30:56.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.605+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c0109b00 con 0x7f89c00f5410 2023-11-15T04:30:56.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.605+0000 7f89beffd700 1 -- 172.21.15.190:0/409631136 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89b80095f0 con 0x7f89c00f5410 2023-11-15T04:30:56.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.605+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c0059100 con 0x7f89c00f5410 2023-11-15T04:30:56.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.606+0000 7f89beffd700 1 --2- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f89a802ffb0 0x7f89a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:56.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.606+0000 7f89beffd700 1 -- 172.21.15.190:0/409631136 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89b8041050 con 0x7f89c00f5410 2023-11-15T04:30:56.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.606+0000 7f89c5a49700 1 --2- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f89a802ffb0 0x7f89a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:56.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.606+0000 7f89c5a49700 1 --2- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f89a802ffb0 0x7f89a8032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f89b40096a0 tx=0x7f89b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:56.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.610+0000 7f89beffd700 1 -- 172.21.15.190:0/409631136 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89b8014420 con 0x7f89c00f5410 2023-11-15T04:30:56.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.816+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89c0045ad0 con 0x7f89c00f5410 2023-11-15T04:30:56.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.817+0000 7f89beffd700 1 -- 172.21.15.190:0/409631136 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f89b801d7f0 con 0x7f89c00f5410 2023-11-15T04:30:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:56.817 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:56.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f89a802ffb0 msgr2=0x7f89a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:56.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 --2- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f89a802ffb0 0x7f89a8032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f89b40096a0 tx=0x7f89b4006b40).stop 2023-11-15T04:30:56.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 msgr2=0x7f89c0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:56.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 --2- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 0x7f89c0109450 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f89b8013510 tx=0x7f89b800b040).stop 2023-11-15T04:30:56.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 shutdown_connections 2023-11-15T04:30:56.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 --2- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f89a802ffb0 0x7f89a8032470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:56.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 --2- 172.21.15.190:0/409631136 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89c00f5410 0x7f89c0109450 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:56.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 >> 172.21.15.190:0/409631136 conn(0x7f89c00f0b70 msgr2=0x7f89c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:56.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 shutdown_connections 2023-11-15T04:30:56.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:56.819+0000 7f89c84ae700 1 -- 172.21.15.190:0/409631136 wait complete. 2023-11-15T04:30:56.821 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:30:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:56.817564+0000 mon.smithi047 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.190:0/409631136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:30:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:30:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:57.071307+0000 mgr.smithi047.zgysuz (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:30:58.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:30:58.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:30:59.274 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.273+0000 7f0b5ba33700 1 -- 172.21.15.190:0/3632039757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 msgr2=0x7f0b540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:59.274 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.273+0000 7f0b5ba33700 1 --2- 172.21.15.190:0/3632039757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 0x7f0b540f6930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f0b44009fb0 tx=0x7f0b44009b30).stop 2023-11-15T04:30:59.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.273+0000 7f0b5ba33700 1 -- 172.21.15.190:0/3632039757 shutdown_connections 2023-11-15T04:30:59.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.273+0000 7f0b5ba33700 1 --2- 172.21.15.190:0/3632039757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 0x7f0b540f6930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:59.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.273+0000 7f0b5ba33700 1 -- 172.21.15.190:0/3632039757 >> 172.21.15.190:0/3632039757 conn(0x7f0b540f1c70 msgr2=0x7f0b540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:59.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.273+0000 7f0b5ba33700 1 -- 172.21.15.190:0/3632039757 shutdown_connections 2023-11-15T04:30:59.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.273+0000 7f0b5ba33700 1 -- 172.21.15.190:0/3632039757 wait complete. 2023-11-15T04:30:59.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.273+0000 7f0b5ba33700 1 Processor -- start 2023-11-15T04:30:59.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.273+0000 7f0b5ba33700 1 -- start start 2023-11-15T04:30:59.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.274+0000 7f0b5ba33700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 0x7f0b54106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:59.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.274+0000 7f0b5ba33700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b54106c20 con 0x7f0b540f6510 2023-11-15T04:30:59.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.274+0000 7f0b597cf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 0x7f0b54106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:59.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.274+0000 7f0b597cf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 0x7f0b54106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:44020/0 (socket says 172.21.15.190:44020) 2023-11-15T04:30:59.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.274+0000 7f0b597cf700 1 -- 172.21.15.190:0/4260136415 learned_addr learned my addr 172.21.15.190:0/4260136415 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:30:59.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.274+0000 7f0b597cf700 1 -- 172.21.15.190:0/4260136415 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b44005040 con 0x7f0b540f6510 2023-11-15T04:30:59.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.274+0000 7f0b597cf700 1 --2- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 0x7f0b54106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f0b44013510 tx=0x7f0b4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:59.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.275+0000 7f0b49ffb700 1 -- 172.21.15.190:0/4260136415 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b44009320 con 0x7f0b540f6510 2023-11-15T04:30:59.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.275+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b54106e20 con 0x7f0b540f6510 2023-11-15T04:30:59.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.275+0000 7f0b49ffb700 1 -- 172.21.15.190:0/4260136415 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b44013f00 con 0x7f0b540f6510 2023-11-15T04:30:59.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.275+0000 7f0b49ffb700 1 -- 172.21.15.190:0/4260136415 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b44014d90 con 0x7f0b540f6510 2023-11-15T04:30:59.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.275+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b541072b0 con 0x7f0b540f6510 2023-11-15T04:30:59.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.276+0000 7f0b49ffb700 1 -- 172.21.15.190:0/4260136415 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b440095f0 con 0x7f0b540f6510 2023-11-15T04:30:59.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.276+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b54045ad0 con 0x7f0b540f6510 2023-11-15T04:30:59.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.276+0000 7f0b49ffb700 1 --2- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0b4002ff60 0x7f0b40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:30:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.276+0000 7f0b49ffb700 1 -- 172.21.15.190:0/4260136415 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b440410a0 con 0x7f0b540f6510 2023-11-15T04:30:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.276+0000 7f0b58fce700 1 --2- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0b4002ff60 0x7f0b40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:30:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.277+0000 7f0b58fce700 1 --2- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0b4002ff60 0x7f0b40032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0b500096a0 tx=0x7f0b50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:30:59.283 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.281+0000 7f0b49ffb700 1 -- 172.21.15.190:0/4260136415 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b4401d7f0 con 0x7f0b540f6510 2023-11-15T04:30:59.489 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.488+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b540fbf60 con 0x7f0b540f6510 2023-11-15T04:30:59.489 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.488+0000 7f0b49ffb700 1 -- 172.21.15.190:0/4260136415 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b44010230 con 0x7f0b540f6510 2023-11-15T04:30:59.490 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:30:59.490 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:30:59.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0b4002ff60 msgr2=0x7f0b40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:59.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 --2- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0b4002ff60 0x7f0b40032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0b500096a0 tx=0x7f0b50006b40).stop 2023-11-15T04:30:59.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 msgr2=0x7f0b54106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:30:59.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 --2- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 0x7f0b54106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f0b44013510 tx=0x7f0b4400b040).stop 2023-11-15T04:30:59.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 shutdown_connections 2023-11-15T04:30:59.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 --2- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0b4002ff60 0x7f0b40032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:59.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 --2- 172.21.15.190:0/4260136415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b540f6510 0x7f0b54106730 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:30:59.493 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 >> 172.21.15.190:0/4260136415 conn(0x7f0b540f1c70 msgr2=0x7f0b540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:30:59.493 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 shutdown_connections 2023-11-15T04:30:59.493 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:30:59.490+0000 7f0b5ba33700 1 -- 172.21.15.190:0/4260136415 wait complete. 2023-11-15T04:30:59.494 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:30:59.071615+0000 mgr.smithi047.zgysuz (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:30:59.489996+0000 mon.smithi047 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.190:0/4260136415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:00.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:00.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:02.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.015+0000 7f7a46d50700 1 -- 172.21.15.190:0/1818286575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a400f51a0 msgr2=0x7f7a400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:02.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.015+0000 7f7a46d50700 1 --2- 172.21.15.190:0/1818286575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a400f51a0 0x7f7a400f55c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f7a30009fb0 tx=0x7f7a30009b30).stop 2023-11-15T04:31:02.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.015+0000 7f7a46d50700 1 -- 172.21.15.190:0/1818286575 shutdown_connections 2023-11-15T04:31:02.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.015+0000 7f7a46d50700 1 --2- 172.21.15.190:0/1818286575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a400f51a0 0x7f7a400f55c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:02.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.015+0000 7f7a46d50700 1 -- 172.21.15.190:0/1818286575 >> 172.21.15.190:0/1818286575 conn(0x7f7a400f09a0 msgr2=0x7f7a400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:02.023 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.015+0000 7f7a46d50700 1 -- 172.21.15.190:0/1818286575 shutdown_connections 2023-11-15T04:31:02.023 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.015+0000 7f7a46d50700 1 -- 172.21.15.190:0/1818286575 wait complete. 2023-11-15T04:31:02.023 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.015+0000 7f7a46d50700 1 Processor -- start 2023-11-15T04:31:02.024 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.016+0000 7f7a46d50700 1 -- start start 2023-11-15T04:31:02.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.016+0000 7f7a46d50700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a40105530 0x7f7a40105950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:02.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.016+0000 7f7a46d50700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a40105e40 con 0x7f7a40105530 2023-11-15T04:31:02.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.016+0000 7f7a45d4e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a40105530 0x7f7a40105950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:02.026 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.016+0000 7f7a45d4e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a40105530 0x7f7a40105950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49094/0 (socket says 172.21.15.190:49094) 2023-11-15T04:31:02.026 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.016+0000 7f7a45d4e700 1 -- 172.21.15.190:0/3318572093 learned_addr learned my addr 172.21.15.190:0/3318572093 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:02.027 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.016+0000 7f7a45d4e700 1 -- 172.21.15.190:0/3318572093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a30005040 con 0x7f7a40105530 2023-11-15T04:31:02.027 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.016+0000 7f7a45d4e700 1 --2- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a40105530 0x7f7a40105950 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7a30009f80 tx=0x7f7a30009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:02.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.017+0000 7f7a367fc700 1 -- 172.21.15.190:0/3318572093 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a300136c0 con 0x7f7a40105530 2023-11-15T04:31:02.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.017+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a40106040 con 0x7f7a40105530 2023-11-15T04:31:02.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.017+0000 7f7a367fc700 1 -- 172.21.15.190:0/3318572093 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a30013e70 con 0x7f7a40105530 2023-11-15T04:31:02.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.017+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a40106b90 con 0x7f7a40105530 2023-11-15T04:31:02.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.017+0000 7f7a367fc700 1 -- 172.21.15.190:0/3318572093 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a30014420 con 0x7f7a40105530 2023-11-15T04:31:02.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.017+0000 7f7a367fc700 1 -- 172.21.15.190:0/3318572093 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a30025060 con 0x7f7a40105530 2023-11-15T04:31:02.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.017+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a40059100 con 0x7f7a40105530 2023-11-15T04:31:02.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.018+0000 7f7a367fc700 1 --2- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a2c02ff60 0x7f7a2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:02.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.018+0000 7f7a367fc700 1 -- 172.21.15.190:0/3318572093 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a30040f60 con 0x7f7a40105530 2023-11-15T04:31:02.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.018+0000 7f7a4554d700 1 --2- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a2c02ff60 0x7f7a2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:02.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.018+0000 7f7a4554d700 1 --2- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a2c02ff60 0x7f7a2c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f7a3c0096a0 tx=0x7f7a3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:02.033 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.023+0000 7f7a367fc700 1 -- 172.21.15.190:0/3318572093 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a3006d2b0 con 0x7f7a40105530 2023-11-15T04:31:02.230 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.229+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a40107350 con 0x7f7a40105530 2023-11-15T04:31:02.231 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.230+0000 7f7a367fc700 1 -- 172.21.15.190:0/3318572093 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a3002d2d0 con 0x7f7a40105530 2023-11-15T04:31:02.231 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:02.232 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:02.234 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a2c02ff60 msgr2=0x7f7a2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:02.234 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 --2- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a2c02ff60 0x7f7a2c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f7a3c0096a0 tx=0x7f7a3c006b40).stop 2023-11-15T04:31:02.234 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a40105530 msgr2=0x7f7a40105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:02.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 --2- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a40105530 0x7f7a40105950 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7a30009f80 tx=0x7f7a30009b30).stop 2023-11-15T04:31:02.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 shutdown_connections 2023-11-15T04:31:02.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 --2- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7a2c02ff60 0x7f7a2c032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:02.236 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 --2- 172.21.15.190:0/3318572093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a40105530 0x7f7a40105950 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:02.236 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 >> 172.21.15.190:0/3318572093 conn(0x7f7a400f09a0 msgr2=0x7f7a400f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:02.237 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 shutdown_connections 2023-11-15T04:31:02.237 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:02.231+0000 7f7a46d50700 1 -- 172.21.15.190:0/3318572093 wait complete. 2023-11-15T04:31:02.237 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:01.071983+0000 mgr.smithi047.zgysuz (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:03.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:03 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:02.231083+0000 mon.smithi047 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.190:0/3318572093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:03.615 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:03.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:03.072300+0000 mgr.smithi047.zgysuz (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:04.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.730+0000 7f4219c65700 1 -- 172.21.15.190:0/606404039 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 msgr2=0x7f42140f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:04.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.730+0000 7f4219c65700 1 --2- 172.21.15.190:0/606404039 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 0x7f42140f5bd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4208009fb0 tx=0x7f4208009b30).stop 2023-11-15T04:31:04.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.730+0000 7f4219c65700 1 -- 172.21.15.190:0/606404039 shutdown_connections 2023-11-15T04:31:04.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.730+0000 7f4219c65700 1 --2- 172.21.15.190:0/606404039 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 0x7f42140f5bd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:04.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.730+0000 7f4219c65700 1 -- 172.21.15.190:0/606404039 >> 172.21.15.190:0/606404039 conn(0x7f42140f0120 msgr2=0x7f42140f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:04.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.730+0000 7f4219c65700 1 -- 172.21.15.190:0/606404039 shutdown_connections 2023-11-15T04:31:04.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.730+0000 7f4219c65700 1 -- 172.21.15.190:0/606404039 wait complete. 2023-11-15T04:31:04.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.730+0000 7f4219c65700 1 Processor -- start 2023-11-15T04:31:04.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.730+0000 7f4219c65700 1 -- start start 2023-11-15T04:31:04.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.731+0000 7f4219c65700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 0x7f42140fd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:04.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.731+0000 7f4219c65700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42140fb590 con 0x7f42140f57b0 2023-11-15T04:31:04.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.731+0000 7f4218c63700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 0x7f42140fd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:04.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.731+0000 7f4218c63700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 0x7f42140fd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49116/0 (socket says 172.21.15.190:49116) 2023-11-15T04:31:04.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.731+0000 7f4218c63700 1 -- 172.21.15.190:0/1482815959 learned_addr learned my addr 172.21.15.190:0/1482815959 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:04.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.731+0000 7f4218c63700 1 -- 172.21.15.190:0/1482815959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4208005040 con 0x7f42140f57b0 2023-11-15T04:31:04.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.732+0000 7f4218c63700 1 --2- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 0x7f42140fd110 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f4208013510 tx=0x7f420800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:04.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.732+0000 7f42117fa700 1 -- 172.21.15.190:0/1482815959 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4208009320 con 0x7f42140f57b0 2023-11-15T04:31:04.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.732+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42140fb730 con 0x7f42140f57b0 2023-11-15T04:31:04.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.732+0000 7f42117fa700 1 -- 172.21.15.190:0/1482815959 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4208013f00 con 0x7f42140f57b0 2023-11-15T04:31:04.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.732+0000 7f42117fa700 1 -- 172.21.15.190:0/1482815959 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4208014dc0 con 0x7f42140f57b0 2023-11-15T04:31:04.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.732+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42140fbbc0 con 0x7f42140f57b0 2023-11-15T04:31:04.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.733+0000 7f42117fa700 1 -- 172.21.15.190:0/1482815959 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42080095f0 con 0x7f42140f57b0 2023-11-15T04:31:04.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.733+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4214059100 con 0x7f42140f57b0 2023-11-15T04:31:04.738 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.733+0000 7f42117fa700 1 --2- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f420402ff60 0x7f4204032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:04.738 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.733+0000 7f42117fa700 1 -- 172.21.15.190:0/1482815959 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42080410f0 con 0x7f42140f57b0 2023-11-15T04:31:04.738 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.733+0000 7f4213fff700 1 --2- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f420402ff60 0x7f4204032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:04.738 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.733+0000 7f4213fff700 1 --2- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f420402ff60 0x7f4204032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f42000096a0 tx=0x7f4200006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:04.739 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.738+0000 7f42117fa700 1 -- 172.21.15.190:0/1482815959 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4208014420 con 0x7f42140f57b0 2023-11-15T04:31:04.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.944+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42140fabd0 con 0x7f42140f57b0 2023-11-15T04:31:04.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.944+0000 7f42117fa700 1 -- 172.21.15.190:0/1482815959 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4208010230 con 0x7f42140f57b0 2023-11-15T04:31:04.947 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:04.947 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:04.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f420402ff60 msgr2=0x7f4204032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:04.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 --2- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f420402ff60 0x7f4204032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f42000096a0 tx=0x7f4200006b40).stop 2023-11-15T04:31:04.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 msgr2=0x7f42140fd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:04.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 --2- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 0x7f42140fd110 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f4208013510 tx=0x7f420800b040).stop 2023-11-15T04:31:04.954 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 shutdown_connections 2023-11-15T04:31:04.955 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 --2- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f420402ff60 0x7f4204032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:04.955 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 --2- 172.21.15.190:0/1482815959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42140f57b0 0x7f42140fd110 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:04.956 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 >> 172.21.15.190:0/1482815959 conn(0x7f42140f0120 msgr2=0x7f42140f4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:04.956 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 shutdown_connections 2023-11-15T04:31:04.956 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:04.946+0000 7f4219c65700 1 -- 172.21.15.190:0/1482815959 wait complete. 2023-11-15T04:31:04.957 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:05.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:05 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:04.946033+0000 mon.smithi047 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.190:0/1482815959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:06.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:06.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:05.072678+0000 mgr.smithi047.zgysuz (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:05.214939+0000 mon.smithi047 (mon.0) 153 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:31:06.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:05.217810+0000 mon.smithi047 (mon.0) 154 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:31:07.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.397+0000 7f221e939700 1 -- 172.21.15.190:0/3656551447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22180f6510 msgr2=0x7f22180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:07.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.397+0000 7f221e939700 1 --2- 172.21.15.190:0/3656551447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22180f6510 0x7f22180f6930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f220c009fb0 tx=0x7f220c009b30).stop 2023-11-15T04:31:07.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.397+0000 7f221e939700 1 -- 172.21.15.190:0/3656551447 shutdown_connections 2023-11-15T04:31:07.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.397+0000 7f221e939700 1 --2- 172.21.15.190:0/3656551447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22180f6510 0x7f22180f6930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:07.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.397+0000 7f221e939700 1 -- 172.21.15.190:0/3656551447 >> 172.21.15.190:0/3656551447 conn(0x7f22180f1c70 msgr2=0x7f22180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:07.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.397+0000 7f221e939700 1 -- 172.21.15.190:0/3656551447 shutdown_connections 2023-11-15T04:31:07.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.397+0000 7f221e939700 1 -- 172.21.15.190:0/3656551447 wait complete. 2023-11-15T04:31:07.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.397+0000 7f221e939700 1 Processor -- start 2023-11-15T04:31:07.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.398+0000 7f221e939700 1 -- start start 2023-11-15T04:31:07.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.398+0000 7f221e939700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2218106840 0x7f2218106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:07.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.398+0000 7f221e939700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2218107150 con 0x7f2218106840 2023-11-15T04:31:07.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.398+0000 7f2217fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2218106840 0x7f2218106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:07.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.398+0000 7f2217fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2218106840 0x7f2218106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49124/0 (socket says 172.21.15.190:49124) 2023-11-15T04:31:07.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.398+0000 7f2217fff700 1 -- 172.21.15.190:0/4126371411 learned_addr learned my addr 172.21.15.190:0/4126371411 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:07.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.398+0000 7f2217fff700 1 -- 172.21.15.190:0/4126371411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c005040 con 0x7f2218106840 2023-11-15T04:31:07.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.399+0000 7f2217fff700 1 --2- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2218106840 0x7f2218106c60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f220c009f80 tx=0x7f220c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.399+0000 7f2214ff9700 1 -- 172.21.15.190:0/4126371411 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f220c0136c0 con 0x7f2218106840 2023-11-15T04:31:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.399+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2218107350 con 0x7f2218106840 2023-11-15T04:31:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.399+0000 7f2214ff9700 1 -- 172.21.15.190:0/4126371411 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f220c0095f0 con 0x7f2218106840 2023-11-15T04:31:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.399+0000 7f2214ff9700 1 -- 172.21.15.190:0/4126371411 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f220c014420 con 0x7f2218106840 2023-11-15T04:31:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.399+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2218107ea0 con 0x7f2218106840 2023-11-15T04:31:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.400+0000 7f2214ff9700 1 -- 172.21.15.190:0/4126371411 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f220c025060 con 0x7f2218106840 2023-11-15T04:31:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.400+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2218059100 con 0x7f2218106840 2023-11-15T04:31:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.400+0000 7f2214ff9700 1 --2- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f220002ff60 0x7f2200032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.400+0000 7f2214ff9700 1 -- 172.21.15.190:0/4126371411 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f220c040d80 con 0x7f2218106840 2023-11-15T04:31:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.400+0000 7f22177fe700 1 --2- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f220002ff60 0x7f2200032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.400+0000 7f22177fe700 1 --2- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f220002ff60 0x7f2200032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f22080096a0 tx=0x7f2208006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.405+0000 7f2214ff9700 1 -- 172.21.15.190:0/4126371411 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f220c01cb60 con 0x7f2218106840 2023-11-15T04:31:07.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.612+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2218108660 con 0x7f2218106840 2023-11-15T04:31:07.615 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.612+0000 7f2214ff9700 1 -- 172.21.15.190:0/4126371411 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f220c042070 con 0x7f2218106840 2023-11-15T04:31:07.615 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:07.615 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:07.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f220002ff60 msgr2=0x7f2200032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:07.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 --2- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f220002ff60 0x7f2200032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f22080096a0 tx=0x7f2208006b40).stop 2023-11-15T04:31:07.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2218106840 msgr2=0x7f2218106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:07.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 --2- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2218106840 0x7f2218106c60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f220c009f80 tx=0x7f220c009b30).stop 2023-11-15T04:31:07.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 shutdown_connections 2023-11-15T04:31:07.617 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 --2- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f220002ff60 0x7f2200032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:07.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 --2- 172.21.15.190:0/4126371411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2218106840 0x7f2218106c60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:07.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 >> 172.21.15.190:0/4126371411 conn(0x7f22180f1c70 msgr2=0x7f22180f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:07.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 shutdown_connections 2023-11-15T04:31:07.618 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:07.614+0000 7f221e939700 1 -- 172.21.15.190:0/4126371411 wait complete. 2023-11-15T04:31:07.618 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:08.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:07.072998+0000 mgr.smithi047.zgysuz (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:08.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:07.613882+0000 mon.smithi047 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.190:0/4126371411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:08.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:08.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:10.129 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.126+0000 7fe99887f700 1 -- 172.21.15.190:0/2127853934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 msgr2=0x7fe9940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:10.129 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.126+0000 7fe99887f700 1 --2- 172.21.15.190:0/2127853934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 0x7fe9940f55c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fe984009fb0 tx=0x7fe984009b30).stop 2023-11-15T04:31:10.129 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.126+0000 7fe99887f700 1 -- 172.21.15.190:0/2127853934 shutdown_connections 2023-11-15T04:31:10.129 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.126+0000 7fe99887f700 1 --2- 172.21.15.190:0/2127853934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 0x7fe9940f55c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:10.130 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.126+0000 7fe99887f700 1 -- 172.21.15.190:0/2127853934 >> 172.21.15.190:0/2127853934 conn(0x7fe9940f09a0 msgr2=0x7fe9940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:10.130 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.126+0000 7fe99887f700 1 -- 172.21.15.190:0/2127853934 shutdown_connections 2023-11-15T04:31:10.130 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.126+0000 7fe99887f700 1 -- 172.21.15.190:0/2127853934 wait complete. 2023-11-15T04:31:10.130 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.126+0000 7fe99887f700 1 Processor -- start 2023-11-15T04:31:10.130 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe99887f700 1 -- start start 2023-11-15T04:31:10.131 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe99887f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 0x7fe9941091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:10.131 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe99887f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe994060240 con 0x7fe9940f51a0 2023-11-15T04:31:10.131 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe9937fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 0x7fe9941091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:10.131 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe9937fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 0x7fe9941091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49142/0 (socket says 172.21.15.190:49142) 2023-11-15T04:31:10.131 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe9937fe700 1 -- 172.21.15.190:0/1826235214 learned_addr learned my addr 172.21.15.190:0/1826235214 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:10.132 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe9937fe700 1 -- 172.21.15.190:0/1826235214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe984005040 con 0x7fe9940f51a0 2023-11-15T04:31:10.132 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe9937fe700 1 --2- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 0x7fe9941091b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe984013510 tx=0x7fe98400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:10.132 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe97bfff700 1 -- 172.21.15.190:0/1826235214 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe984009320 con 0x7fe9940f51a0 2023-11-15T04:31:10.132 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe994060440 con 0x7fe9940f51a0 2023-11-15T04:31:10.132 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.127+0000 7fe97bfff700 1 -- 172.21.15.190:0/1826235214 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe984013f00 con 0x7fe9940f51a0 2023-11-15T04:31:10.132 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.128+0000 7fe97bfff700 1 -- 172.21.15.190:0/1826235214 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe984014dc0 con 0x7fe9940f51a0 2023-11-15T04:31:10.133 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.128+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe994109860 con 0x7fe9940f51a0 2023-11-15T04:31:10.134 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.128+0000 7fe97bfff700 1 -- 172.21.15.190:0/1826235214 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9840095f0 con 0x7fe9940f51a0 2023-11-15T04:31:10.134 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.128+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe994045ad0 con 0x7fe9940f51a0 2023-11-15T04:31:10.134 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.128+0000 7fe97bfff700 1 --2- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe97c02ff60 0x7fe97c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:10.134 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.128+0000 7fe97bfff700 1 -- 172.21.15.190:0/1826235214 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9840410c0 con 0x7fe9940f51a0 2023-11-15T04:31:10.135 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.129+0000 7fe992ffd700 1 --2- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe97c02ff60 0x7fe97c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:10.135 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.129+0000 7fe992ffd700 1 --2- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe97c02ff60 0x7fe97c032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe9880096a0 tx=0x7fe988006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:10.135 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.134+0000 7fe97bfff700 1 -- 172.21.15.190:0/1826235214 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe98401c790 con 0x7fe9940f51a0 2023-11-15T04:31:10.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.339+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe99410a080 con 0x7fe9940f51a0 2023-11-15T04:31:10.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.339+0000 7fe97bfff700 1 -- 172.21.15.190:0/1826235214 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe984010230 con 0x7fe9940f51a0 2023-11-15T04:31:10.341 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:10.341 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:10.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe97c02ff60 msgr2=0x7fe97c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:10.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 --2- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe97c02ff60 0x7fe97c032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe9880096a0 tx=0x7fe988006b40).stop 2023-11-15T04:31:10.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 msgr2=0x7fe9941091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:10.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 --2- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 0x7fe9941091b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe984013510 tx=0x7fe98400b040).stop 2023-11-15T04:31:10.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 shutdown_connections 2023-11-15T04:31:10.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 --2- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe97c02ff60 0x7fe97c032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:10.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 --2- 172.21.15.190:0/1826235214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9940f51a0 0x7fe9941091b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:10.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 >> 172.21.15.190:0/1826235214 conn(0x7fe9940f09a0 msgr2=0x7fe9940f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:10.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 shutdown_connections 2023-11-15T04:31:10.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:10.341+0000 7fe99887f700 1 -- 172.21.15.190:0/1826235214 wait complete. 2023-11-15T04:31:10.344 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:10.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:09.073312+0000 mgr.smithi047.zgysuz (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:11.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:10.341008+0000 mon.smithi047 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.190:0/1826235214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:11.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:11.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:12.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:12 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:11.073603+0000 mgr.smithi047.zgysuz (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:12.876 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.874+0000 7f07dc8c4700 1 -- 172.21.15.190:0/4125924178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 msgr2=0x7f07d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:12.877 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.874+0000 7f07dc8c4700 1 --2- 172.21.15.190:0/4125924178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 0x7f07d80f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f07c8009fb0 tx=0x7f07c8009b30).stop 2023-11-15T04:31:12.877 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.874+0000 7f07dc8c4700 1 -- 172.21.15.190:0/4125924178 shutdown_connections 2023-11-15T04:31:12.877 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.874+0000 7f07dc8c4700 1 --2- 172.21.15.190:0/4125924178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 0x7f07d80f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:12.877 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.874+0000 7f07dc8c4700 1 -- 172.21.15.190:0/4125924178 >> 172.21.15.190:0/4125924178 conn(0x7f07d80f0b70 msgr2=0x7f07d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:12.878 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.874+0000 7f07dc8c4700 1 -- 172.21.15.190:0/4125924178 shutdown_connections 2023-11-15T04:31:12.878 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.874+0000 7f07dc8c4700 1 -- 172.21.15.190:0/4125924178 wait complete. 2023-11-15T04:31:12.878 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.874+0000 7f07dc8c4700 1 Processor -- start 2023-11-15T04:31:12.878 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.874+0000 7f07dc8c4700 1 -- start start 2023-11-15T04:31:12.878 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.875+0000 7f07dc8c4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 0x7f07d81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:12.879 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.875+0000 7f07dc8c4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8060280 con 0x7f07d80f5410 2023-11-15T04:31:12.879 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.875+0000 7f07d659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 0x7f07d81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:12.879 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.875+0000 7f07d659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 0x7f07d81093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41752/0 (socket says 172.21.15.190:41752) 2023-11-15T04:31:12.879 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.875+0000 7f07d659c700 1 -- 172.21.15.190:0/1723793206 learned_addr learned my addr 172.21.15.190:0/1723793206 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:12.879 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.875+0000 7f07d659c700 1 -- 172.21.15.190:0/1723793206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c8005040 con 0x7f07d80f5410 2023-11-15T04:31:12.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.875+0000 7f07d659c700 1 --2- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 0x7f07d81093f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f07c8013510 tx=0x7f07c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:12.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.875+0000 7f07c6ffd700 1 -- 172.21.15.190:0/1723793206 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c8009320 con 0x7f07d80f5410 2023-11-15T04:31:12.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.876+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d8060480 con 0x7f07d80f5410 2023-11-15T04:31:12.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.876+0000 7f07c6ffd700 1 -- 172.21.15.190:0/1723793206 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07c8013f00 con 0x7f07d80f5410 2023-11-15T04:31:12.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.876+0000 7f07c6ffd700 1 -- 172.21.15.190:0/1723793206 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c8014d90 con 0x7f07d80f5410 2023-11-15T04:31:12.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.876+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d8109b20 con 0x7f07d80f5410 2023-11-15T04:31:12.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.876+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d8045ad0 con 0x7f07d80f5410 2023-11-15T04:31:12.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.876+0000 7f07c6ffd700 1 -- 172.21.15.190:0/1723793206 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07c80095f0 con 0x7f07d80f5410 2023-11-15T04:31:12.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.876+0000 7f07c6ffd700 1 --2- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ff60 0x7f07c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:12.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.877+0000 7f07c6ffd700 1 -- 172.21.15.190:0/1723793206 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07c8040fc0 con 0x7f07d80f5410 2023-11-15T04:31:12.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.877+0000 7f07d5d9b700 1 --2- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ff60 0x7f07c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:12.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.877+0000 7f07d5d9b700 1 --2- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ff60 0x7f07c0032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f07cc0096a0 tx=0x7f07cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:12.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:12.881+0000 7f07c6ffd700 1 -- 172.21.15.190:0/1723793206 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07c8014420 con 0x7f07d80f5410 2023-11-15T04:31:13.089 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.089+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07d810a370 con 0x7f07d80f5410 2023-11-15T04:31:13.090 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.089+0000 7f07c6ffd700 1 -- 172.21.15.190:0/1723793206 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07c8010230 con 0x7f07d80f5410 2023-11-15T04:31:13.090 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:13.091 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:13.092 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ff60 msgr2=0x7f07c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:13.092 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 --2- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ff60 0x7f07c0032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f07cc0096a0 tx=0x7f07cc006b40).stop 2023-11-15T04:31:13.092 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 msgr2=0x7f07d81093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:13.092 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 --2- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 0x7f07d81093f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f07c8013510 tx=0x7f07c800b040).stop 2023-11-15T04:31:13.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 shutdown_connections 2023-11-15T04:31:13.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 --2- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ff60 0x7f07c0032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:13.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 --2- 172.21.15.190:0/1723793206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d80f5410 0x7f07d81093f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:13.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 >> 172.21.15.190:0/1723793206 conn(0x7f07d80f0b70 msgr2=0x7f07d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:13.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 shutdown_connections 2023-11-15T04:31:13.094 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:13.091+0000 7f07dc8c4700 1 -- 172.21.15.190:0/1723793206 wait complete. 2023-11-15T04:31:13.094 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:13.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:13 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:13.090998+0000 mon.smithi047 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.190:0/1723793206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:14.462 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:14.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:13.073977+0000 mgr.smithi047.zgysuz (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:15.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 -- 172.21.15.190:0/4080698687 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 msgr2=0x7f39a80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:15.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 --2- 172.21.15.190:0/4080698687 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 0x7f39a80f66c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f3998009fb0 tx=0x7f3998009b30).stop 2023-11-15T04:31:15.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 -- 172.21.15.190:0/4080698687 shutdown_connections 2023-11-15T04:31:15.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 --2- 172.21.15.190:0/4080698687 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 0x7f39a80f66c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:15.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 -- 172.21.15.190:0/4080698687 >> 172.21.15.190:0/4080698687 conn(0x7f39a80f1aa0 msgr2=0x7f39a80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:15.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 -- 172.21.15.190:0/4080698687 shutdown_connections 2023-11-15T04:31:15.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 -- 172.21.15.190:0/4080698687 wait complete. 2023-11-15T04:31:15.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 Processor -- start 2023-11-15T04:31:15.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 -- start start 2023-11-15T04:31:15.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.599+0000 7f39ad33f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 0x7f39a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:15.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39ad33f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a8069e20 con 0x7f39a80f62a0 2023-11-15T04:31:15.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39a7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 0x7f39a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:15.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39a7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 0x7f39a806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41766/0 (socket says 172.21.15.190:41766) 2023-11-15T04:31:15.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39a7fff700 1 -- 172.21.15.190:0/1894610575 learned_addr learned my addr 172.21.15.190:0/1894610575 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:15.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39a7fff700 1 -- 172.21.15.190:0/1894610575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3998005040 con 0x7f39a80f62a0 2023-11-15T04:31:15.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39a7fff700 1 --2- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 0x7f39a806b9a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f3998013510 tx=0x7f399800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:15.607 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39a4ff9700 1 -- 172.21.15.190:0/1894610575 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3998009320 con 0x7f39a80f62a0 2023-11-15T04:31:15.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a8069fc0 con 0x7f39a80f62a0 2023-11-15T04:31:15.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39a4ff9700 1 -- 172.21.15.190:0/1894610575 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3998013f00 con 0x7f39a80f62a0 2023-11-15T04:31:15.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39a4ff9700 1 -- 172.21.15.190:0/1894610575 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3998014dc0 con 0x7f39a80f62a0 2023-11-15T04:31:15.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.600+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a806a450 con 0x7f39a80f62a0 2023-11-15T04:31:15.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.601+0000 7f39a4ff9700 1 -- 172.21.15.190:0/1894610575 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39980095f0 con 0x7f39a80f62a0 2023-11-15T04:31:15.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.601+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a8059100 con 0x7f39a80f62a0 2023-11-15T04:31:15.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.601+0000 7f39a4ff9700 1 --2- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f399002ff60 0x7f3990032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:15.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.601+0000 7f39a4ff9700 1 -- 172.21.15.190:0/1894610575 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39980410c0 con 0x7f39a80f62a0 2023-11-15T04:31:15.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.602+0000 7f39a77fe700 1 --2- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f399002ff60 0x7f3990032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:15.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.602+0000 7f39a77fe700 1 --2- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f399002ff60 0x7f3990032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f399c0096a0 tx=0x7f399c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:15.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.606+0000 7f39a4ff9700 1 -- 172.21.15.190:0/1894610575 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3998014420 con 0x7f39a80f62a0 2023-11-15T04:31:15.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.815+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39a80fbce0 con 0x7f39a80f62a0 2023-11-15T04:31:15.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.815+0000 7f39a4ff9700 1 -- 172.21.15.190:0/1894610575 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3998010230 con 0x7f39a80f62a0 2023-11-15T04:31:15.816 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:15.817 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:15.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f399002ff60 msgr2=0x7f3990032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:15.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 --2- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f399002ff60 0x7f3990032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f399c0096a0 tx=0x7f399c006b40).stop 2023-11-15T04:31:15.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 msgr2=0x7f39a806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:15.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 --2- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 0x7f39a806b9a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f3998013510 tx=0x7f399800b040).stop 2023-11-15T04:31:15.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 shutdown_connections 2023-11-15T04:31:15.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 --2- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f399002ff60 0x7f3990032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:15.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 --2- 172.21.15.190:0/1894610575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39a80f62a0 0x7f39a806b9a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:15.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 >> 172.21.15.190:0/1894610575 conn(0x7f39a80f1aa0 msgr2=0x7f39a80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:15.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 shutdown_connections 2023-11-15T04:31:15.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:15.817+0000 7f39ad33f700 1 -- 172.21.15.190:0/1894610575 wait complete. 2023-11-15T04:31:15.820 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:15.074345+0000 mgr.smithi047.zgysuz (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:15.816683+0000 mon.smithi047 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.190:0/1894610575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:17.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:17.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:18.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:17.074747+0000 mgr.smithi047.zgysuz (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:18.355 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.353+0000 7f2b907cb700 1 -- 172.21.15.190:0/3162227094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 msgr2=0x7f2b880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:18.355 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.353+0000 7f2b907cb700 1 --2- 172.21.15.190:0/3162227094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 0x7f2b880f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2b78009fb0 tx=0x7f2b78009b30).stop 2023-11-15T04:31:18.355 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.353+0000 7f2b907cb700 1 -- 172.21.15.190:0/3162227094 shutdown_connections 2023-11-15T04:31:18.356 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.353+0000 7f2b907cb700 1 --2- 172.21.15.190:0/3162227094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 0x7f2b880f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:18.356 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.353+0000 7f2b907cb700 1 -- 172.21.15.190:0/3162227094 >> 172.21.15.190:0/3162227094 conn(0x7f2b880f1c70 msgr2=0x7f2b880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:18.356 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.353+0000 7f2b907cb700 1 -- 172.21.15.190:0/3162227094 shutdown_connections 2023-11-15T04:31:18.356 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.353+0000 7f2b907cb700 1 -- 172.21.15.190:0/3162227094 wait complete. 2023-11-15T04:31:18.356 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.354+0000 7f2b907cb700 1 Processor -- start 2023-11-15T04:31:18.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.354+0000 7f2b907cb700 1 -- start start 2023-11-15T04:31:18.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.354+0000 7f2b907cb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 0x7f2b88106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:18.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.354+0000 7f2b907cb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b88106c20 con 0x7f2b880f6510 2023-11-15T04:31:18.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.354+0000 7f2b8e567700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 0x7f2b88106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:18.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.354+0000 7f2b8e567700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 0x7f2b88106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41774/0 (socket says 172.21.15.190:41774) 2023-11-15T04:31:18.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.354+0000 7f2b8e567700 1 -- 172.21.15.190:0/2722077008 learned_addr learned my addr 172.21.15.190:0/2722077008 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:18.362 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.354+0000 7f2b8e567700 1 -- 172.21.15.190:0/2722077008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b78005040 con 0x7f2b880f6510 2023-11-15T04:31:18.362 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.354+0000 7f2b8e567700 1 --2- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 0x7f2b88106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2b78013510 tx=0x7f2b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:18.362 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.355+0000 7f2b7effd700 1 -- 172.21.15.190:0/2722077008 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b78009320 con 0x7f2b880f6510 2023-11-15T04:31:18.362 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.355+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b88106e20 con 0x7f2b880f6510 2023-11-15T04:31:18.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.355+0000 7f2b7effd700 1 -- 172.21.15.190:0/2722077008 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b78013f00 con 0x7f2b880f6510 2023-11-15T04:31:18.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.355+0000 7f2b7effd700 1 -- 172.21.15.190:0/2722077008 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b78014d90 con 0x7f2b880f6510 2023-11-15T04:31:18.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.355+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b881072b0 con 0x7f2b880f6510 2023-11-15T04:31:18.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.355+0000 7f2b7effd700 1 -- 172.21.15.190:0/2722077008 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b78014420 con 0x7f2b880f6510 2023-11-15T04:31:18.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.356+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b88045ad0 con 0x7f2b880f6510 2023-11-15T04:31:18.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.356+0000 7f2b7effd700 1 --2- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2b7402ff60 0x7f2b74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:18.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.356+0000 7f2b7effd700 1 -- 172.21.15.190:0/2722077008 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b780410c0 con 0x7f2b880f6510 2023-11-15T04:31:18.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.356+0000 7f2b8dd66700 1 --2- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2b7402ff60 0x7f2b74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:18.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.356+0000 7f2b8dd66700 1 --2- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2b7402ff60 0x7f2b74032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f2b840096a0 tx=0x7f2b84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.361+0000 7f2b7effd700 1 -- 172.21.15.190:0/2722077008 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b780095f0 con 0x7f2b880f6510 2023-11-15T04:31:18.570 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.569+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b880fbf60 con 0x7f2b880f6510 2023-11-15T04:31:18.571 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.570+0000 7f2b7effd700 1 -- 172.21.15.190:0/2722077008 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b7801db70 con 0x7f2b880f6510 2023-11-15T04:31:18.571 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:18.572 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:18.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2b7402ff60 msgr2=0x7f2b74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:18.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 --2- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2b7402ff60 0x7f2b74032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f2b840096a0 tx=0x7f2b84006b40).stop 2023-11-15T04:31:18.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 msgr2=0x7f2b88106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:18.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 --2- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 0x7f2b88106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2b78013510 tx=0x7f2b7800b040).stop 2023-11-15T04:31:18.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 shutdown_connections 2023-11-15T04:31:18.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 --2- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2b7402ff60 0x7f2b74032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:18.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 --2- 172.21.15.190:0/2722077008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b880f6510 0x7f2b88106730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:18.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 >> 172.21.15.190:0/2722077008 conn(0x7f2b880f1c70 msgr2=0x7f2b880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:18.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 shutdown_connections 2023-11-15T04:31:18.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:18.572+0000 7f2b907cb700 1 -- 172.21.15.190:0/2722077008 wait complete. 2023-11-15T04:31:18.575 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:19.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:18.571815+0000 mon.smithi047 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.190:0/2722077008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:19.967 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:19.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:20.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:19 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:19.075205+0000 mgr.smithi047.zgysuz (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:21.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.040+0000 7f8ad2057700 1 -- 172.21.15.190:0/2326231334 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 msgr2=0x7f8acc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.040+0000 7f8ad2057700 1 --2- 172.21.15.190:0/2326231334 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 0x7f8acc0f55c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f8ac8009fb0 tx=0x7f8ac8009b30).stop 2023-11-15T04:31:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.040+0000 7f8ad2057700 1 -- 172.21.15.190:0/2326231334 shutdown_connections 2023-11-15T04:31:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.040+0000 7f8ad2057700 1 --2- 172.21.15.190:0/2326231334 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 0x7f8acc0f55c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.040+0000 7f8ad2057700 1 -- 172.21.15.190:0/2326231334 >> 172.21.15.190:0/2326231334 conn(0x7f8acc0f09a0 msgr2=0x7f8acc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.040+0000 7f8ad2057700 1 -- 172.21.15.190:0/2326231334 shutdown_connections 2023-11-15T04:31:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.040+0000 7f8ad2057700 1 -- 172.21.15.190:0/2326231334 wait complete. 2023-11-15T04:31:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.041+0000 7f8ad2057700 1 Processor -- start 2023-11-15T04:31:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.041+0000 7f8ad2057700 1 -- start start 2023-11-15T04:31:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.041+0000 7f8ad2057700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 0x7f8acc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.041+0000 7f8ad2057700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc060240 con 0x7f8acc0f51a0 2023-11-15T04:31:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.041+0000 7f8ad1055700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 0x7f8acc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.041+0000 7f8ad1055700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 0x7f8acc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:36694/0 (socket says 172.21.15.190:36694) 2023-11-15T04:31:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.041+0000 7f8ad1055700 1 -- 172.21.15.190:0/2396680473 learned_addr learned my addr 172.21.15.190:0/2396680473 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.041+0000 7f8ad1055700 1 -- 172.21.15.190:0/2396680473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac8005040 con 0x7f8acc0f51a0 2023-11-15T04:31:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.042+0000 7f8ad1055700 1 --2- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 0x7f8acc1091b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8ac8013510 tx=0x7f8ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:21.047 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.042+0000 7f8ac1ffb700 1 -- 172.21.15.190:0/2396680473 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ac8009320 con 0x7f8acc0f51a0 2023-11-15T04:31:21.047 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.042+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8acc060440 con 0x7f8acc0f51a0 2023-11-15T04:31:21.047 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.042+0000 7f8ac1ffb700 1 -- 172.21.15.190:0/2396680473 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ac8013f00 con 0x7f8acc0f51a0 2023-11-15T04:31:21.047 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.042+0000 7f8ac1ffb700 1 -- 172.21.15.190:0/2396680473 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ac8014dc0 con 0x7f8acc0f51a0 2023-11-15T04:31:21.048 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.042+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8acc109860 con 0x7f8acc0f51a0 2023-11-15T04:31:21.048 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.043+0000 7f8ac1ffb700 1 -- 172.21.15.190:0/2396680473 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ac80095f0 con 0x7f8acc0f51a0 2023-11-15T04:31:21.048 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.043+0000 7f8ac1ffb700 1 --2- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8ab802ff60 0x7f8ab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:21.048 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.043+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8acc045ad0 con 0x7f8acc0f51a0 2023-11-15T04:31:21.048 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.043+0000 7f8ac1ffb700 1 -- 172.21.15.190:0/2396680473 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ac80410e0 con 0x7f8acc0f51a0 2023-11-15T04:31:21.049 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.043+0000 7f8ad0854700 1 --2- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8ab802ff60 0x7f8ab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:21.049 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.044+0000 7f8ad0854700 1 --2- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8ab802ff60 0x7f8ab8032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8abc0096a0 tx=0x7f8abc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:21.049 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.048+0000 7f8ac1ffb700 1 -- 172.21.15.190:0/2396680473 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ac8014420 con 0x7f8acc0f51a0 2023-11-15T04:31:21.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.252+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8acc10a080 con 0x7f8acc0f51a0 2023-11-15T04:31:21.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.253+0000 7f8ac1ffb700 1 -- 172.21.15.190:0/2396680473 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ac801c920 con 0x7f8acc0f51a0 2023-11-15T04:31:21.255 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:21.255 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:21.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8ab802ff60 msgr2=0x7f8ab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:21.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 --2- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8ab802ff60 0x7f8ab8032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8abc0096a0 tx=0x7f8abc006b40).stop 2023-11-15T04:31:21.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 msgr2=0x7f8acc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:21.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 --2- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 0x7f8acc1091b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8ac8013510 tx=0x7f8ac800b040).stop 2023-11-15T04:31:21.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 shutdown_connections 2023-11-15T04:31:21.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 --2- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8ab802ff60 0x7f8ab8032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:21.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 --2- 172.21.15.190:0/2396680473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8acc0f51a0 0x7f8acc1091b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:21.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 >> 172.21.15.190:0/2396680473 conn(0x7f8acc0f09a0 msgr2=0x7f8acc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:21.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 shutdown_connections 2023-11-15T04:31:21.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:21.255+0000 7f8ad2057700 1 -- 172.21.15.190:0/2396680473 wait complete. 2023-11-15T04:31:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:21.075619+0000 mgr.smithi047.zgysuz (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:21.254599+0000 mon.smithi047 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.190:0/2396680473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:22.676 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:22.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:23.814 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.812+0000 7fe8fd21f700 1 -- 172.21.15.190:0/238019368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 msgr2=0x7fe8f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:23.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.812+0000 7fe8fd21f700 1 --2- 172.21.15.190:0/238019368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 0x7fe8f80f90f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe8e8009fb0 tx=0x7fe8e8009b30).stop 2023-11-15T04:31:23.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.813+0000 7fe8fd21f700 1 -- 172.21.15.190:0/238019368 shutdown_connections 2023-11-15T04:31:23.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.813+0000 7fe8fd21f700 1 --2- 172.21.15.190:0/238019368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 0x7fe8f80f90f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:23.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.813+0000 7fe8fd21f700 1 -- 172.21.15.190:0/238019368 >> 172.21.15.190:0/238019368 conn(0x7fe8f80f0b70 msgr2=0x7fe8f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:23.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.813+0000 7fe8fd21f700 1 -- 172.21.15.190:0/238019368 shutdown_connections 2023-11-15T04:31:23.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.813+0000 7fe8fd21f700 1 -- 172.21.15.190:0/238019368 wait complete. 2023-11-15T04:31:23.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.813+0000 7fe8fd21f700 1 Processor -- start 2023-11-15T04:31:23.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.813+0000 7fe8fd21f700 1 -- start start 2023-11-15T04:31:23.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.813+0000 7fe8fd21f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 0x7fe8f81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:23.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.813+0000 7fe8fd21f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f8106bd0 con 0x7fe8f8060280 2023-11-15T04:31:23.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8f6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 0x7fe8f81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:23.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8f6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 0x7fe8f81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:36712/0 (socket says 172.21.15.190:36712) 2023-11-15T04:31:23.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8f6d9d700 1 -- 172.21.15.190:0/3629773422 learned_addr learned my addr 172.21.15.190:0/3629773422 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:23.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8f6d9d700 1 -- 172.21.15.190:0/3629773422 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e8005040 con 0x7fe8f8060280 2023-11-15T04:31:23.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8f6d9d700 1 --2- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 0x7fe8f81066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fe8e8013510 tx=0x7fe8e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:23.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8df7fe700 1 -- 172.21.15.190:0/3629773422 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e8009320 con 0x7fe8f8060280 2023-11-15T04:31:23.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f8106dd0 con 0x7fe8f8060280 2023-11-15T04:31:23.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f8107260 con 0x7fe8f8060280 2023-11-15T04:31:23.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8df7fe700 1 -- 172.21.15.190:0/3629773422 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8e8013f00 con 0x7fe8f8060280 2023-11-15T04:31:23.821 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.814+0000 7fe8df7fe700 1 -- 172.21.15.190:0/3629773422 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e8014d90 con 0x7fe8f8060280 2023-11-15T04:31:23.821 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.815+0000 7fe8df7fe700 1 -- 172.21.15.190:0/3629773422 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8e80095f0 con 0x7fe8f8060280 2023-11-15T04:31:23.821 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.815+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d8004fa0 con 0x7fe8f8060280 2023-11-15T04:31:23.821 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.815+0000 7fe8df7fe700 1 --2- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe8e002ff60 0x7fe8e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:23.822 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.815+0000 7fe8df7fe700 1 -- 172.21.15.190:0/3629773422 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8e8040f70 con 0x7fe8f8060280 2023-11-15T04:31:23.822 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.816+0000 7fe8f659c700 1 --2- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe8e002ff60 0x7fe8e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:23.822 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.816+0000 7fe8f659c700 1 --2- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe8e002ff60 0x7fe8e0032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe8ec0096a0 tx=0x7fe8ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:23.823 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:23.820+0000 7fe8df7fe700 1 -- 172.21.15.190:0/3629773422 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8e806d2b0 con 0x7fe8f8060280 2023-11-15T04:31:24.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.025+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8d8005670 con 0x7fe8f8060280 2023-11-15T04:31:24.026 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.025+0000 7fe8df7fe700 1 -- 172.21.15.190:0/3629773422 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8e8014420 con 0x7fe8f8060280 2023-11-15T04:31:24.027 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:24.027 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:24.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe8e002ff60 msgr2=0x7fe8e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:24.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 --2- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe8e002ff60 0x7fe8e0032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe8ec0096a0 tx=0x7fe8ec006b40).stop 2023-11-15T04:31:24.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 msgr2=0x7fe8f81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:24.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 --2- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 0x7fe8f81066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fe8e8013510 tx=0x7fe8e800b040).stop 2023-11-15T04:31:24.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 shutdown_connections 2023-11-15T04:31:24.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 --2- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe8e002ff60 0x7fe8e0032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:24.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 --2- 172.21.15.190:0/3629773422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f8060280 0x7fe8f81066e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:24.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 >> 172.21.15.190:0/3629773422 conn(0x7fe8f80f0b70 msgr2=0x7fe8f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:24.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 shutdown_connections 2023-11-15T04:31:24.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:24.027+0000 7fe8fd21f700 1 -- 172.21.15.190:0/3629773422 wait complete. 2023-11-15T04:31:24.030 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:24.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:23.076074+0000 mgr.smithi047.zgysuz (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:24.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:24.026738+0000 mon.smithi047 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.190:0/3629773422' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:25.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:25.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:26.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:25.076402+0000 mgr.smithi047.zgysuz (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:26.513 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.510+0000 7f25b34da700 1 -- 172.21.15.190:0/1753395387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 msgr2=0x7f25ac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:26.513 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.510+0000 7f25b34da700 1 --2- 172.21.15.190:0/1753395387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 0x7f25ac0f90f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f25a0009fb0 tx=0x7f25a0009b30).stop 2023-11-15T04:31:26.513 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.510+0000 7f25b34da700 1 -- 172.21.15.190:0/1753395387 shutdown_connections 2023-11-15T04:31:26.513 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.510+0000 7f25b34da700 1 --2- 172.21.15.190:0/1753395387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 0x7f25ac0f90f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:26.514 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.510+0000 7f25b34da700 1 -- 172.21.15.190:0/1753395387 >> 172.21.15.190:0/1753395387 conn(0x7f25ac0f0b70 msgr2=0x7f25ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:26.514 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.510+0000 7f25b34da700 1 -- 172.21.15.190:0/1753395387 shutdown_connections 2023-11-15T04:31:26.514 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.510+0000 7f25b34da700 1 -- 172.21.15.190:0/1753395387 wait complete. 2023-11-15T04:31:26.514 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.511+0000 7f25b34da700 1 Processor -- start 2023-11-15T04:31:26.514 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.511+0000 7f25b34da700 1 -- start start 2023-11-15T04:31:26.515 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.511+0000 7f25b34da700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 0x7f25ac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:26.515 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.511+0000 7f25b34da700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac106bd0 con 0x7f25ac060280 2023-11-15T04:31:26.515 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.511+0000 7f25b1276700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 0x7f25ac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:26.515 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.511+0000 7f25b1276700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 0x7f25ac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:36726/0 (socket says 172.21.15.190:36726) 2023-11-15T04:31:26.515 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.511+0000 7f25b1276700 1 -- 172.21.15.190:0/1208586015 learned_addr learned my addr 172.21.15.190:0/1208586015 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:26.516 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.511+0000 7f25b1276700 1 -- 172.21.15.190:0/1208586015 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0005040 con 0x7f25ac060280 2023-11-15T04:31:26.516 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.511+0000 7f25b1276700 1 --2- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 0x7f25ac1066e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f25a0013510 tx=0x7f25a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:26.516 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.512+0000 7f259dffb700 1 -- 172.21.15.190:0/1208586015 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25a0009320 con 0x7f25ac060280 2023-11-15T04:31:26.516 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.512+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac106dd0 con 0x7f25ac060280 2023-11-15T04:31:26.516 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.512+0000 7f259dffb700 1 -- 172.21.15.190:0/1208586015 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25a0013f00 con 0x7f25ac060280 2023-11-15T04:31:26.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.512+0000 7f259dffb700 1 -- 172.21.15.190:0/1208586015 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25a0014dc0 con 0x7f25ac060280 2023-11-15T04:31:26.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.512+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac107260 con 0x7f25ac060280 2023-11-15T04:31:26.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.513+0000 7f259dffb700 1 -- 172.21.15.190:0/1208586015 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25a00095f0 con 0x7f25ac060280 2023-11-15T04:31:26.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.513+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac045ad0 con 0x7f25ac060280 2023-11-15T04:31:26.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.513+0000 7f259dffb700 1 --2- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f259802ff60 0x7f2598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:26.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.513+0000 7f259dffb700 1 -- 172.21.15.190:0/1208586015 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25a0041020 con 0x7f25ac060280 2023-11-15T04:31:26.519 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.513+0000 7f25b0a75700 1 --2- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f259802ff60 0x7f2598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:26.519 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.513+0000 7f25b0a75700 1 --2- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f259802ff60 0x7f2598032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f25a80096a0 tx=0x7f25a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:26.519 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.517+0000 7f259dffb700 1 -- 172.21.15.190:0/1208586015 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25a0014420 con 0x7f25ac060280 2023-11-15T04:31:26.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.722+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25ac1080b0 con 0x7f25ac060280 2023-11-15T04:31:26.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.722+0000 7f259dffb700 1 -- 172.21.15.190:0/1208586015 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f25a001c920 con 0x7f25ac060280 2023-11-15T04:31:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:26.724 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:26.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f259802ff60 msgr2=0x7f2598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:26.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 --2- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f259802ff60 0x7f2598032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f25a80096a0 tx=0x7f25a8006b40).stop 2023-11-15T04:31:26.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 msgr2=0x7f25ac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:26.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 --2- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 0x7f25ac1066e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f25a0013510 tx=0x7f25a000b040).stop 2023-11-15T04:31:26.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 shutdown_connections 2023-11-15T04:31:26.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 --2- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f259802ff60 0x7f2598032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:26.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 --2- 172.21.15.190:0/1208586015 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac060280 0x7f25ac1066e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:26.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 >> 172.21.15.190:0/1208586015 conn(0x7f25ac0f0b70 msgr2=0x7f25ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:26.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 shutdown_connections 2023-11-15T04:31:26.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:26.724+0000 7f25b34da700 1 -- 172.21.15.190:0/1208586015 wait complete. 2023-11-15T04:31:26.727 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:26.723903+0000 mon.smithi047 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.190:0/1208586015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:27.076828+0000 mgr.smithi047.zgysuz (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:28.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:28.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:29.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.263+0000 7f273d44a700 1 -- 172.21.15.190:0/3500288154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 msgr2=0x7f27380f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:29.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.263+0000 7f273d44a700 1 --2- 172.21.15.190:0/3500288154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 0x7f27380f7f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f2728009fb0 tx=0x7f2728009b30).stop 2023-11-15T04:31:29.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.263+0000 7f273d44a700 1 -- 172.21.15.190:0/3500288154 shutdown_connections 2023-11-15T04:31:29.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.263+0000 7f273d44a700 1 --2- 172.21.15.190:0/3500288154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 0x7f27380f7f00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:29.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.263+0000 7f273d44a700 1 -- 172.21.15.190:0/3500288154 >> 172.21.15.190:0/3500288154 conn(0x7f27380f0260 msgr2=0x7f27380f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:29.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.264+0000 7f273d44a700 1 -- 172.21.15.190:0/3500288154 shutdown_connections 2023-11-15T04:31:29.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.264+0000 7f273d44a700 1 -- 172.21.15.190:0/3500288154 wait complete. 2023-11-15T04:31:29.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.264+0000 7f273d44a700 1 Processor -- start 2023-11-15T04:31:29.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.264+0000 7f273d44a700 1 -- start start 2023-11-15T04:31:29.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.264+0000 7f273d44a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 0x7f2738106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:29.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.264+0000 7f273d44a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2738106c00 con 0x7f27380f7ae0 2023-11-15T04:31:29.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f2736ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 0x7f2738106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:29.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f2736ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 0x7f2738106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:36744/0 (socket says 172.21.15.190:36744) 2023-11-15T04:31:29.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f2736ffd700 1 -- 172.21.15.190:0/2767983865 learned_addr learned my addr 172.21.15.190:0/2767983865 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:29.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f2736ffd700 1 -- 172.21.15.190:0/2767983865 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2728005040 con 0x7f27380f7ae0 2023-11-15T04:31:29.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f2736ffd700 1 --2- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 0x7f2738106710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f2728013510 tx=0x7f272800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:29.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f271f7fe700 1 -- 172.21.15.190:0/2767983865 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2728009320 con 0x7f27380f7ae0 2023-11-15T04:31:29.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2738106e00 con 0x7f27380f7ae0 2023-11-15T04:31:29.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f271f7fe700 1 -- 172.21.15.190:0/2767983865 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2728013f00 con 0x7f27380f7ae0 2023-11-15T04:31:29.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f271f7fe700 1 -- 172.21.15.190:0/2767983865 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2728014d90 con 0x7f27380f7ae0 2023-11-15T04:31:29.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.265+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2738107290 con 0x7f27380f7ae0 2023-11-15T04:31:29.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.266+0000 7f271f7fe700 1 -- 172.21.15.190:0/2767983865 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2728014420 con 0x7f27380f7ae0 2023-11-15T04:31:29.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.266+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2738045ad0 con 0x7f27380f7ae0 2023-11-15T04:31:29.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.266+0000 7f271f7fe700 1 --2- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f272002ff60 0x7f2720032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:29.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.266+0000 7f271f7fe700 1 -- 172.21.15.190:0/2767983865 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27280410a0 con 0x7f27380f7ae0 2023-11-15T04:31:29.272 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.267+0000 7f27367fc700 1 --2- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f272002ff60 0x7f2720032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:29.272 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.267+0000 7f27367fc700 1 --2- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f272002ff60 0x7f2720032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f272c0096a0 tx=0x7f272c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:29.272 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.271+0000 7f271f7fe700 1 -- 172.21.15.190:0/2767983865 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27280095f0 con 0x7f27380f7ae0 2023-11-15T04:31:29.478 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.477+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27380fbfc0 con 0x7f27380f7ae0 2023-11-15T04:31:29.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.478+0000 7f271f7fe700 1 -- 172.21.15.190:0/2767983865 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f272801db70 con 0x7f27380f7ae0 2023-11-15T04:31:29.479 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:29.479 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:29.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f272002ff60 msgr2=0x7f2720032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:29.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 --2- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f272002ff60 0x7f2720032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f272c0096a0 tx=0x7f272c006b40).stop 2023-11-15T04:31:29.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 msgr2=0x7f2738106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:29.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 --2- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 0x7f2738106710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f2728013510 tx=0x7f272800b040).stop 2023-11-15T04:31:29.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 shutdown_connections 2023-11-15T04:31:29.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 --2- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f272002ff60 0x7f2720032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:29.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 --2- 172.21.15.190:0/2767983865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27380f7ae0 0x7f2738106710 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:29.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 >> 172.21.15.190:0/2767983865 conn(0x7f27380f0260 msgr2=0x7f27380f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:29.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 shutdown_connections 2023-11-15T04:31:29.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:29.480+0000 7f273d44a700 1 -- 172.21.15.190:0/2767983865 wait complete. 2023-11-15T04:31:29.483 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:30.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:29.077274+0000 mgr.smithi047.zgysuz (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:30.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:29.479466+0000 mon.smithi047 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.190:0/2767983865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:30.864 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:30.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:32.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.998+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2568072226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 msgr2=0x7fa1440f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:32.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.998+0000 7fa14ba5f700 1 --2- 172.21.15.190:0/2568072226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 0x7fa1440f68a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fa134009fb0 tx=0x7fa134009b30).stop 2023-11-15T04:31:32.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.998+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2568072226 shutdown_connections 2023-11-15T04:31:32.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.998+0000 7fa14ba5f700 1 --2- 172.21.15.190:0/2568072226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 0x7fa1440f68a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:32.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.998+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2568072226 >> 172.21.15.190:0/2568072226 conn(0x7fa1440f1c00 msgr2=0x7fa1440f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:32.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.998+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2568072226 shutdown_connections 2023-11-15T04:31:32.002 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.998+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2568072226 wait complete. 2023-11-15T04:31:32.002 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.998+0000 7fa14ba5f700 1 Processor -- start 2023-11-15T04:31:32.002 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.998+0000 7fa14ba5f700 1 -- start start 2023-11-15T04:31:32.002 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa14ba5f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 0x7fa1441066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:32.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa14ba5f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa144106bc0 con 0x7fa1440f6480 2023-11-15T04:31:32.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa1497fb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 0x7fa1441066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:32.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa1497fb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 0x7fa1441066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53866/0 (socket says 172.21.15.190:53866) 2023-11-15T04:31:32.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa1497fb700 1 -- 172.21.15.190:0/2653507121 learned_addr learned my addr 172.21.15.190:0/2653507121 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:32.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa1497fb700 1 -- 172.21.15.190:0/2653507121 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa134005040 con 0x7fa1440f6480 2023-11-15T04:31:32.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa1497fb700 1 --2- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 0x7fa1441066d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa134013510 tx=0x7fa13400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:32.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa139ffb700 1 -- 172.21.15.190:0/2653507121 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa134009320 con 0x7fa1440f6480 2023-11-15T04:31:32.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa144106dc0 con 0x7fa1440f6480 2023-11-15T04:31:32.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa139ffb700 1 -- 172.21.15.190:0/2653507121 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa134013f00 con 0x7fa1440f6480 2023-11-15T04:31:32.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:31.999+0000 7fa139ffb700 1 -- 172.21.15.190:0/2653507121 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa134014d90 con 0x7fa1440f6480 2023-11-15T04:31:32.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.000+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa144107250 con 0x7fa1440f6480 2023-11-15T04:31:32.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.001+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa144045ad0 con 0x7fa1440f6480 2023-11-15T04:31:32.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.001+0000 7fa139ffb700 1 -- 172.21.15.190:0/2653507121 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1340095f0 con 0x7fa1440f6480 2023-11-15T04:31:32.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.001+0000 7fa139ffb700 1 --2- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa13002ff60 0x7fa130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:32.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.001+0000 7fa139ffb700 1 -- 172.21.15.190:0/2653507121 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa134041000 con 0x7fa1440f6480 2023-11-15T04:31:32.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.001+0000 7fa148ffa700 1 --2- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa13002ff60 0x7fa130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:32.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.002+0000 7fa148ffa700 1 --2- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa13002ff60 0x7fa130032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa1400096a0 tx=0x7fa140006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:32.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.006+0000 7fa139ffb700 1 -- 172.21.15.190:0/2653507121 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa13401d7f0 con 0x7fa1440f6480 2023-11-15T04:31:32.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.212+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa144108100 con 0x7fa1440f6480 2023-11-15T04:31:32.214 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.213+0000 7fa139ffb700 1 -- 172.21.15.190:0/2653507121 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa134010230 con 0x7fa1440f6480 2023-11-15T04:31:32.215 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:32.215 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:32.216 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa13002ff60 msgr2=0x7fa130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:32.216 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 --2- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa13002ff60 0x7fa130032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa1400096a0 tx=0x7fa140006b40).stop 2023-11-15T04:31:32.216 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 msgr2=0x7fa1441066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:32.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 --2- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 0x7fa1441066d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa134013510 tx=0x7fa13400b040).stop 2023-11-15T04:31:32.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 shutdown_connections 2023-11-15T04:31:32.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 --2- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa13002ff60 0x7fa130032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:32.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 --2- 172.21.15.190:0/2653507121 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1440f6480 0x7fa1441066d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:32.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 >> 172.21.15.190:0/2653507121 conn(0x7fa1440f1c00 msgr2=0x7fa1440f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:32.218 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 shutdown_connections 2023-11-15T04:31:32.218 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:32.215+0000 7fa14ba5f700 1 -- 172.21.15.190:0/2653507121 wait complete. 2023-11-15T04:31:32.218 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:31.077690+0000 mgr.smithi047.zgysuz (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:33.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:33 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:32.214383+0000 mon.smithi047 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.190:0/2653507121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:33.605 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:33.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:34.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:33.078118+0000 mgr.smithi047.zgysuz (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:34.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 -- 172.21.15.190:0/86824666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 msgr2=0x7f89580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:34.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 --2- 172.21.15.190:0/86824666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 0x7f89580f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f894c009fb0 tx=0x7f894c009b30).stop 2023-11-15T04:31:34.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 -- 172.21.15.190:0/86824666 shutdown_connections 2023-11-15T04:31:34.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 --2- 172.21.15.190:0/86824666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 0x7f89580f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:34.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 -- 172.21.15.190:0/86824666 >> 172.21.15.190:0/86824666 conn(0x7f89580f0b70 msgr2=0x7f89580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:34.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 -- 172.21.15.190:0/86824666 shutdown_connections 2023-11-15T04:31:34.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 -- 172.21.15.190:0/86824666 wait complete. 2023-11-15T04:31:34.758 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 Processor -- start 2023-11-15T04:31:34.758 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 -- start start 2023-11-15T04:31:34.758 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 0x7f8958109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:34.758 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.755+0000 7f895fedb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8958060220 con 0x7f89580f5410 2023-11-15T04:31:34.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f895dc77700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 0x7f8958109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:34.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f895dc77700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 0x7f8958109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53874/0 (socket says 172.21.15.190:53874) 2023-11-15T04:31:34.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f895dc77700 1 -- 172.21.15.190:0/464276315 learned_addr learned my addr 172.21.15.190:0/464276315 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:34.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f895dc77700 1 -- 172.21.15.190:0/464276315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c005040 con 0x7f89580f5410 2023-11-15T04:31:34.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f895dc77700 1 --2- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 0x7f8958109450 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f894c013510 tx=0x7f894c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:34.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f894a7fc700 1 -- 172.21.15.190:0/464276315 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f894c009320 con 0x7f89580f5410 2023-11-15T04:31:34.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8958060420 con 0x7f89580f5410 2023-11-15T04:31:34.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f894a7fc700 1 -- 172.21.15.190:0/464276315 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f894c013f00 con 0x7f89580f5410 2023-11-15T04:31:34.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f894a7fc700 1 -- 172.21.15.190:0/464276315 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f894c014d90 con 0x7f89580f5410 2023-11-15T04:31:34.762 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.756+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8958109b00 con 0x7f89580f5410 2023-11-15T04:31:34.762 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.757+0000 7f894a7fc700 1 -- 172.21.15.190:0/464276315 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f894c0095f0 con 0x7f89580f5410 2023-11-15T04:31:34.762 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.757+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8958059100 con 0x7f89580f5410 2023-11-15T04:31:34.762 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.757+0000 7f894a7fc700 1 --2- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f894402ffb0 0x7f8944032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:34.762 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.757+0000 7f894a7fc700 1 -- 172.21.15.190:0/464276315 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f894c0410a0 con 0x7f89580f5410 2023-11-15T04:31:34.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.758+0000 7f895d476700 1 --2- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f894402ffb0 0x7f8944032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:34.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.758+0000 7f895d476700 1 --2- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f894402ffb0 0x7f8944032470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f89540096a0 tx=0x7f8954006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:34.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.762+0000 7f894a7fc700 1 -- 172.21.15.190:0/464276315 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f894c06d2b0 con 0x7f89580f5410 2023-11-15T04:31:34.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.967+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8958045ad0 con 0x7f89580f5410 2023-11-15T04:31:34.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.967+0000 7f894a7fc700 1 -- 172.21.15.190:0/464276315 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f894c014420 con 0x7f89580f5410 2023-11-15T04:31:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:34.969 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:34.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f894402ffb0 msgr2=0x7f8944032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:34.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 --2- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f894402ffb0 0x7f8944032470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f89540096a0 tx=0x7f8954006b40).stop 2023-11-15T04:31:34.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 msgr2=0x7f8958109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:34.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 --2- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 0x7f8958109450 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f894c013510 tx=0x7f894c00b040).stop 2023-11-15T04:31:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 shutdown_connections 2023-11-15T04:31:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 --2- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f894402ffb0 0x7f8944032470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 --2- 172.21.15.190:0/464276315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89580f5410 0x7f8958109450 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 >> 172.21.15.190:0/464276315 conn(0x7f89580f0b70 msgr2=0x7f89580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 shutdown_connections 2023-11-15T04:31:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:34.969+0000 7f895fedb700 1 -- 172.21.15.190:0/464276315 wait complete. 2023-11-15T04:31:34.972 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:35.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:35 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:34.968765+0000 mon.smithi047 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.190:0/464276315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:36.358 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:36.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:35.078436+0000 mgr.smithi047.zgysuz (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:37.445 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.443+0000 7f5d8d07c700 1 -- 172.21.15.190:0/1612179773 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 msgr2=0x7f5d880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:37.446 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.443+0000 7f5d8d07c700 1 --2- 172.21.15.190:0/1612179773 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 0x7f5d880f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f5d70009fb0 tx=0x7f5d70009b30).stop 2023-11-15T04:31:37.446 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.443+0000 7f5d8d07c700 1 -- 172.21.15.190:0/1612179773 shutdown_connections 2023-11-15T04:31:37.446 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.443+0000 7f5d8d07c700 1 --2- 172.21.15.190:0/1612179773 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 0x7f5d880f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:37.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.443+0000 7f5d8d07c700 1 -- 172.21.15.190:0/1612179773 >> 172.21.15.190:0/1612179773 conn(0x7f5d880f1c70 msgr2=0x7f5d880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:37.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.443+0000 7f5d8d07c700 1 -- 172.21.15.190:0/1612179773 shutdown_connections 2023-11-15T04:31:37.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.443+0000 7f5d8d07c700 1 -- 172.21.15.190:0/1612179773 wait complete. 2023-11-15T04:31:37.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.444+0000 7f5d8d07c700 1 Processor -- start 2023-11-15T04:31:37.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.444+0000 7f5d8d07c700 1 -- start start 2023-11-15T04:31:37.448 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.444+0000 7f5d8d07c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 0x7f5d8806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:37.448 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.444+0000 7f5d8d07c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88069e20 con 0x7f5d880f6510 2023-11-15T04:31:37.448 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.444+0000 7f5d86d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 0x7f5d8806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:37.448 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.445+0000 7f5d86d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 0x7f5d8806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53888/0 (socket says 172.21.15.190:53888) 2023-11-15T04:31:37.448 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.445+0000 7f5d86d9d700 1 -- 172.21.15.190:0/505745340 learned_addr learned my addr 172.21.15.190:0/505745340 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:37.449 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.445+0000 7f5d86d9d700 1 -- 172.21.15.190:0/505745340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70005040 con 0x7f5d880f6510 2023-11-15T04:31:37.449 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.445+0000 7f5d86d9d700 1 --2- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 0x7f5d8806b9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f5d70013510 tx=0x7f5d7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:37.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.445+0000 7f5d7f7fe700 1 -- 172.21.15.190:0/505745340 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d70009320 con 0x7f5d880f6510 2023-11-15T04:31:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.445+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d88069fc0 con 0x7f5d880f6510 2023-11-15T04:31:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.445+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8806a450 con 0x7f5d880f6510 2023-11-15T04:31:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.445+0000 7f5d7f7fe700 1 -- 172.21.15.190:0/505745340 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d70013f00 con 0x7f5d880f6510 2023-11-15T04:31:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.445+0000 7f5d7f7fe700 1 -- 172.21.15.190:0/505745340 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d70014d60 con 0x7f5d880f6510 2023-11-15T04:31:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.446+0000 7f5d7f7fe700 1 -- 172.21.15.190:0/505745340 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d70014420 con 0x7f5d880f6510 2023-11-15T04:31:37.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.446+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d88059100 con 0x7f5d880f6510 2023-11-15T04:31:37.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.446+0000 7f5d7f7fe700 1 --2- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5d7402ff60 0x7f5d74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:37.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.446+0000 7f5d7f7fe700 1 -- 172.21.15.190:0/505745340 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d70041390 con 0x7f5d880f6510 2023-11-15T04:31:37.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.446+0000 7f5d8659c700 1 --2- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5d7402ff60 0x7f5d74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:37.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.447+0000 7f5d8659c700 1 --2- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5d7402ff60 0x7f5d74032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5d780096a0 tx=0x7f5d78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:37.453 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.451+0000 7f5d7f7fe700 1 -- 172.21.15.190:0/505745340 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d70012b20 con 0x7f5d880f6510 2023-11-15T04:31:37.659 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.658+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d880fbf60 con 0x7f5d880f6510 2023-11-15T04:31:37.660 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.659+0000 7f5d7f7fe700 1 -- 172.21.15.190:0/505745340 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d70012b20 con 0x7f5d880f6510 2023-11-15T04:31:37.660 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:37.660 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:37.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5d7402ff60 msgr2=0x7f5d74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:37.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 --2- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5d7402ff60 0x7f5d74032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5d780096a0 tx=0x7f5d78006b40).stop 2023-11-15T04:31:37.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 msgr2=0x7f5d8806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:37.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 --2- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 0x7f5d8806b9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f5d70013510 tx=0x7f5d7000b040).stop 2023-11-15T04:31:37.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 shutdown_connections 2023-11-15T04:31:37.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 --2- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5d7402ff60 0x7f5d74032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:37.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 --2- 172.21.15.190:0/505745340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d880f6510 0x7f5d8806b9a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:37.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 >> 172.21.15.190:0/505745340 conn(0x7f5d880f1c70 msgr2=0x7f5d880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:37.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 shutdown_connections 2023-11-15T04:31:37.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:37.661+0000 7f5d8d07c700 1 -- 172.21.15.190:0/505745340 wait complete. 2023-11-15T04:31:37.664 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:38.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:37.078720+0000 mgr.smithi047.zgysuz (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:38.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:37.660491+0000 mon.smithi047 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.190:0/505745340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:39.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:39.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:40.168 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.166+0000 7f19e7f68700 1 -- 172.21.15.190:0/2934231179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 msgr2=0x7f19e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:40.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.166+0000 7f19e7f68700 1 --2- 172.21.15.190:0/2934231179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 0x7f19e00f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f19d0009fb0 tx=0x7f19d0009b30).stop 2023-11-15T04:31:40.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.167+0000 7f19e7f68700 1 -- 172.21.15.190:0/2934231179 shutdown_connections 2023-11-15T04:31:40.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.167+0000 7f19e7f68700 1 --2- 172.21.15.190:0/2934231179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 0x7f19e00f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:40.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.167+0000 7f19e7f68700 1 -- 172.21.15.190:0/2934231179 >> 172.21.15.190:0/2934231179 conn(0x7f19e00f0b70 msgr2=0x7f19e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:40.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.167+0000 7f19e7f68700 1 -- 172.21.15.190:0/2934231179 shutdown_connections 2023-11-15T04:31:40.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.167+0000 7f19e7f68700 1 -- 172.21.15.190:0/2934231179 wait complete. 2023-11-15T04:31:40.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.167+0000 7f19e7f68700 1 Processor -- start 2023-11-15T04:31:40.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.167+0000 7f19e7f68700 1 -- start start 2023-11-15T04:31:40.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.167+0000 7f19e7f68700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 0x7f19e0109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:40.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.167+0000 7f19e7f68700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e0060280 con 0x7f19e00f5410 2023-11-15T04:31:40.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19e5d04700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 0x7f19e0109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:40.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19e5d04700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 0x7f19e0109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53910/0 (socket says 172.21.15.190:53910) 2023-11-15T04:31:40.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19e5d04700 1 -- 172.21.15.190:0/2868487011 learned_addr learned my addr 172.21.15.190:0/2868487011 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:40.172 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19e5d04700 1 -- 172.21.15.190:0/2868487011 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d0005040 con 0x7f19e00f5410 2023-11-15T04:31:40.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19e5d04700 1 --2- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 0x7f19e0109390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f19d0013510 tx=0x7f19d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:40.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19d67fc700 1 -- 172.21.15.190:0/2868487011 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d0009320 con 0x7f19e00f5410 2023-11-15T04:31:40.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e0060480 con 0x7f19e00f5410 2023-11-15T04:31:40.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19d67fc700 1 -- 172.21.15.190:0/2868487011 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19d0013f00 con 0x7f19e00f5410 2023-11-15T04:31:40.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19d67fc700 1 -- 172.21.15.190:0/2868487011 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d0014d90 con 0x7f19e00f5410 2023-11-15T04:31:40.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.168+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e0109ac0 con 0x7f19e00f5410 2023-11-15T04:31:40.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.169+0000 7f19d67fc700 1 -- 172.21.15.190:0/2868487011 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19d00095f0 con 0x7f19e00f5410 2023-11-15T04:31:40.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.169+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e0045ad0 con 0x7f19e00f5410 2023-11-15T04:31:40.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.170+0000 7f19d67fc700 1 --2- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f19cc02ff60 0x7f19cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:40.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.170+0000 7f19d67fc700 1 -- 172.21.15.190:0/2868487011 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f19d0040fc0 con 0x7f19e00f5410 2023-11-15T04:31:40.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.170+0000 7f19e5503700 1 --2- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f19cc02ff60 0x7f19cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:40.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.170+0000 7f19e5503700 1 --2- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f19cc02ff60 0x7f19cc032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f19dc0096a0 tx=0x7f19dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:40.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.174+0000 7f19d67fc700 1 -- 172.21.15.190:0/2868487011 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19d0014420 con 0x7f19e00f5410 2023-11-15T04:31:40.382 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.381+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19e00fae60 con 0x7f19e00f5410 2023-11-15T04:31:40.382 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.382+0000 7f19d67fc700 1 -- 172.21.15.190:0/2868487011 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f19d0010230 con 0x7f19e00f5410 2023-11-15T04:31:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:40.383 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:40.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f19cc02ff60 msgr2=0x7f19cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:40.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 --2- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f19cc02ff60 0x7f19cc032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f19dc0096a0 tx=0x7f19dc006b40).stop 2023-11-15T04:31:40.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 msgr2=0x7f19e0109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:40.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 --2- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 0x7f19e0109390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f19d0013510 tx=0x7f19d000b040).stop 2023-11-15T04:31:40.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 shutdown_connections 2023-11-15T04:31:40.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 --2- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f19cc02ff60 0x7f19cc032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:40.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 --2- 172.21.15.190:0/2868487011 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e00f5410 0x7f19e0109390 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:40.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 >> 172.21.15.190:0/2868487011 conn(0x7f19e00f0b70 msgr2=0x7f19e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:40.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 shutdown_connections 2023-11-15T04:31:40.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:40.383+0000 7f19e7f68700 1 -- 172.21.15.190:0/2868487011 wait complete. 2023-11-15T04:31:40.386 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:40.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:39.079005+0000 mgr.smithi047.zgysuz (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:41.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:41 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:40.383099+0000 mon.smithi047 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.190:0/2868487011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:41.765 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:41.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:42.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:41.079365+0000 mgr.smithi047.zgysuz (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:42.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.877+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1616188479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 msgr2=0x7f5a9c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:42.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.877+0000 7f5aa20bc700 1 --2- 172.21.15.190:0/1616188479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 0x7f5a9c0f9150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f5a90009fb0 tx=0x7f5a90009b30).stop 2023-11-15T04:31:42.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.877+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1616188479 shutdown_connections 2023-11-15T04:31:42.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.877+0000 7f5aa20bc700 1 --2- 172.21.15.190:0/1616188479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 0x7f5a9c0f9150 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:42.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.877+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1616188479 >> 172.21.15.190:0/1616188479 conn(0x7f5a9c0f0b70 msgr2=0x7f5a9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:42.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.877+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1616188479 shutdown_connections 2023-11-15T04:31:42.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.877+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1616188479 wait complete. 2023-11-15T04:31:42.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.878+0000 7f5aa20bc700 1 Processor -- start 2023-11-15T04:31:42.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.878+0000 7f5aa20bc700 1 -- start start 2023-11-15T04:31:42.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.878+0000 7f5aa20bc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 0x7f5a9c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:42.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.878+0000 7f5aa20bc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c106bd0 con 0x7f5a9c060220 2023-11-15T04:31:42.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.878+0000 7f5a9b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 0x7f5a9c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:42.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.878+0000 7f5a9b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 0x7f5a9c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:45564/0 (socket says 172.21.15.190:45564) 2023-11-15T04:31:42.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.878+0000 7f5a9b7fe700 1 -- 172.21.15.190:0/1646117730 learned_addr learned my addr 172.21.15.190:0/1646117730 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:42.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.878+0000 7f5a9b7fe700 1 -- 172.21.15.190:0/1646117730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a90005040 con 0x7f5a9c060220 2023-11-15T04:31:42.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.879+0000 7f5a9b7fe700 1 --2- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 0x7f5a9c1066e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5a90013510 tx=0x7f5a9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:42.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.879+0000 7f5a83fff700 1 -- 172.21.15.190:0/1646117730 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a90009320 con 0x7f5a9c060220 2023-11-15T04:31:42.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.879+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c106dd0 con 0x7f5a9c060220 2023-11-15T04:31:42.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.879+0000 7f5a83fff700 1 -- 172.21.15.190:0/1646117730 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a90013f00 con 0x7f5a9c060220 2023-11-15T04:31:42.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.879+0000 7f5a83fff700 1 -- 172.21.15.190:0/1646117730 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a90014dc0 con 0x7f5a9c060220 2023-11-15T04:31:42.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.879+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9c107260 con 0x7f5a9c060220 2023-11-15T04:31:42.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.880+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9c045ad0 con 0x7f5a9c060220 2023-11-15T04:31:42.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.880+0000 7f5a83fff700 1 -- 172.21.15.190:0/1646117730 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a900095f0 con 0x7f5a9c060220 2023-11-15T04:31:42.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.880+0000 7f5a83fff700 1 --2- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5a8402ff60 0x7f5a84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:42.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.880+0000 7f5a83fff700 1 -- 172.21.15.190:0/1646117730 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a900410c0 con 0x7f5a9c060220 2023-11-15T04:31:42.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.880+0000 7f5a9affd700 1 --2- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5a8402ff60 0x7f5a84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:42.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.880+0000 7f5a9affd700 1 --2- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5a8402ff60 0x7f5a84032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5a8c0096a0 tx=0x7f5a8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:42.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:42.885+0000 7f5a83fff700 1 -- 172.21.15.190:0/1646117730 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a9001c790 con 0x7f5a9c060220 2023-11-15T04:31:43.092 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.091+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a9c108110 con 0x7f5a9c060220 2023-11-15T04:31:43.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.092+0000 7f5a83fff700 1 -- 172.21.15.190:0/1646117730 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a90010230 con 0x7f5a9c060220 2023-11-15T04:31:43.093 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:43.093 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:43.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5a8402ff60 msgr2=0x7f5a84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:43.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 --2- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5a8402ff60 0x7f5a84032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5a8c0096a0 tx=0x7f5a8c006b40).stop 2023-11-15T04:31:43.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 msgr2=0x7f5a9c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:43.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 --2- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 0x7f5a9c1066e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5a90013510 tx=0x7f5a9000b040).stop 2023-11-15T04:31:43.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 shutdown_connections 2023-11-15T04:31:43.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 --2- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f5a8402ff60 0x7f5a84032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:43.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 --2- 172.21.15.190:0/1646117730 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a9c060220 0x7f5a9c1066e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:43.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 >> 172.21.15.190:0/1646117730 conn(0x7f5a9c0f0b70 msgr2=0x7f5a9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:43.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 shutdown_connections 2023-11-15T04:31:43.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:43.094+0000 7f5aa20bc700 1 -- 172.21.15.190:0/1646117730 wait complete. 2023-11-15T04:31:43.097 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:43.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:43 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:43.093612+0000 mon.smithi047 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.190:0/1646117730' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:44.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:44.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:43.079744+0000 mgr.smithi047.zgysuz (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:45.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.607+0000 7f6957fff700 1 -- 172.21.15.190:0/2605187511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 msgr2=0x7f69580f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:45.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.607+0000 7f6957fff700 1 --2- 172.21.15.190:0/2605187511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 0x7f69580f66c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6948009fb0 tx=0x7f6948009b30).stop 2023-11-15T04:31:45.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.607+0000 7f6957fff700 1 -- 172.21.15.190:0/2605187511 shutdown_connections 2023-11-15T04:31:45.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.607+0000 7f6957fff700 1 --2- 172.21.15.190:0/2605187511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 0x7f69580f66c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:45.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.607+0000 7f6957fff700 1 -- 172.21.15.190:0/2605187511 >> 172.21.15.190:0/2605187511 conn(0x7f69580f1aa0 msgr2=0x7f69580f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:45.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.607+0000 7f6957fff700 1 -- 172.21.15.190:0/2605187511 shutdown_connections 2023-11-15T04:31:45.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.607+0000 7f6957fff700 1 -- 172.21.15.190:0/2605187511 wait complete. 2023-11-15T04:31:45.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.607+0000 7f6957fff700 1 Processor -- start 2023-11-15T04:31:45.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.607+0000 7f6957fff700 1 -- start start 2023-11-15T04:31:45.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f6957fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 0x7f695806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:45.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f6957fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6958069e20 con 0x7f69580f62a0 2023-11-15T04:31:45.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f6956ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 0x7f695806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:45.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f6956ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 0x7f695806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:45594/0 (socket says 172.21.15.190:45594) 2023-11-15T04:31:45.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f6956ffd700 1 -- 172.21.15.190:0/630397774 learned_addr learned my addr 172.21.15.190:0/630397774 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:45.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f6956ffd700 1 -- 172.21.15.190:0/630397774 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948005040 con 0x7f69580f62a0 2023-11-15T04:31:45.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f6956ffd700 1 --2- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 0x7f695806b9a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f6948013510 tx=0x7f694800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:45.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f693f7fe700 1 -- 172.21.15.190:0/630397774 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6948009320 con 0x7f69580f62a0 2023-11-15T04:31:45.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6958069fc0 con 0x7f69580f62a0 2023-11-15T04:31:45.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f693f7fe700 1 -- 172.21.15.190:0/630397774 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6948013f00 con 0x7f69580f62a0 2023-11-15T04:31:45.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f693f7fe700 1 -- 172.21.15.190:0/630397774 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6948014dc0 con 0x7f69580f62a0 2023-11-15T04:31:45.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.608+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695806a450 con 0x7f69580f62a0 2023-11-15T04:31:45.615 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.609+0000 7f693f7fe700 1 -- 172.21.15.190:0/630397774 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69480095f0 con 0x7f69580f62a0 2023-11-15T04:31:45.615 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.609+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6958059100 con 0x7f69580f62a0 2023-11-15T04:31:45.615 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.609+0000 7f693f7fe700 1 --2- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f694002ff60 0x7f6940032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:45.615 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.609+0000 7f693f7fe700 1 -- 172.21.15.190:0/630397774 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6948041100 con 0x7f69580f62a0 2023-11-15T04:31:45.615 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.610+0000 7f69567fc700 1 --2- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f694002ff60 0x7f6940032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:45.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.610+0000 7f69567fc700 1 --2- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f694002ff60 0x7f6940032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f694c0096a0 tx=0x7f694c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:45.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.614+0000 7f693f7fe700 1 -- 172.21.15.190:0/630397774 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6948014420 con 0x7f69580f62a0 2023-11-15T04:31:45.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.819+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69580fbce0 con 0x7f69580f62a0 2023-11-15T04:31:45.855 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.820+0000 7f693f7fe700 1 -- 172.21.15.190:0/630397774 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6948010230 con 0x7f69580f62a0 2023-11-15T04:31:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:45.856 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:45.857 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f694002ff60 msgr2=0x7f6940032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:45.857 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 --2- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f694002ff60 0x7f6940032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f694c0096a0 tx=0x7f694c006b40).stop 2023-11-15T04:31:45.857 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 msgr2=0x7f695806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:45.857 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 --2- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 0x7f695806b9a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f6948013510 tx=0x7f694800b040).stop 2023-11-15T04:31:45.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 shutdown_connections 2023-11-15T04:31:45.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 --2- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f694002ff60 0x7f6940032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:45.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 --2- 172.21.15.190:0/630397774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69580f62a0 0x7f695806b9a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:45.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 >> 172.21.15.190:0/630397774 conn(0x7f69580f1aa0 msgr2=0x7f69580f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:45.859 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 shutdown_connections 2023-11-15T04:31:45.859 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:45.822+0000 7f6957fff700 1 -- 172.21.15.190:0/630397774 wait complete. 2023-11-15T04:31:45.859 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:45.080148+0000 mgr.smithi047.zgysuz (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:46.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:45.821693+0000 mon.smithi047 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.190:0/630397774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:47.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:47.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:47.080513+0000 mgr.smithi047.zgysuz (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:48.427 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.425+0000 7f2e38df5700 1 -- 172.21.15.190:0/4144950018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 msgr2=0x7f2e340f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:48.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.425+0000 7f2e38df5700 1 --2- 172.21.15.190:0/4144950018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 0x7f2e340f4be0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f2e28009fb0 tx=0x7f2e28009b30).stop 2023-11-15T04:31:48.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.425+0000 7f2e38df5700 1 -- 172.21.15.190:0/4144950018 shutdown_connections 2023-11-15T04:31:48.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.425+0000 7f2e38df5700 1 --2- 172.21.15.190:0/4144950018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 0x7f2e340f4be0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:48.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e38df5700 1 -- 172.21.15.190:0/4144950018 >> 172.21.15.190:0/4144950018 conn(0x7f2e340f02e0 msgr2=0x7f2e340f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:48.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e38df5700 1 -- 172.21.15.190:0/4144950018 shutdown_connections 2023-11-15T04:31:48.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e38df5700 1 -- 172.21.15.190:0/4144950018 wait complete. 2023-11-15T04:31:48.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e38df5700 1 Processor -- start 2023-11-15T04:31:48.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e38df5700 1 -- start start 2023-11-15T04:31:48.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e38df5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 0x7f2e34106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:48.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e38df5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34106bf0 con 0x7f2e340f47c0 2023-11-15T04:31:48.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 0x7f2e34106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:48.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 0x7f2e34106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:45602/0 (socket says 172.21.15.190:45602) 2023-11-15T04:31:48.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.426+0000 7f2e3259c700 1 -- 172.21.15.190:0/3375866678 learned_addr learned my addr 172.21.15.190:0/3375866678 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:48.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.427+0000 7f2e3259c700 1 -- 172.21.15.190:0/3375866678 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e28005040 con 0x7f2e340f47c0 2023-11-15T04:31:48.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.427+0000 7f2e3259c700 1 --2- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 0x7f2e34106700 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2e28013510 tx=0x7f2e2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:48.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.427+0000 7f2e22ffd700 1 -- 172.21.15.190:0/3375866678 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e28009320 con 0x7f2e340f47c0 2023-11-15T04:31:48.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.427+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e34106df0 con 0x7f2e340f47c0 2023-11-15T04:31:48.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.427+0000 7f2e22ffd700 1 -- 172.21.15.190:0/3375866678 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e28013f00 con 0x7f2e340f47c0 2023-11-15T04:31:48.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.427+0000 7f2e22ffd700 1 -- 172.21.15.190:0/3375866678 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e28014d90 con 0x7f2e340f47c0 2023-11-15T04:31:48.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.427+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e34107280 con 0x7f2e340f47c0 2023-11-15T04:31:48.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.428+0000 7f2e22ffd700 1 -- 172.21.15.190:0/3375866678 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e280095f0 con 0x7f2e340f47c0 2023-11-15T04:31:48.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.428+0000 7f2e22ffd700 1 --2- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e1c02ff60 0x7f2e1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:48.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.428+0000 7f2e22ffd700 1 -- 172.21.15.190:0/3375866678 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e28041000 con 0x7f2e340f47c0 2023-11-15T04:31:48.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.428+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e34045ad0 con 0x7f2e340f47c0 2023-11-15T04:31:48.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.428+0000 7f2e31d9b700 1 --2- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e1c02ff60 0x7f2e1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:48.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.429+0000 7f2e31d9b700 1 --2- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e1c02ff60 0x7f2e1c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2e240096a0 tx=0x7f2e24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:48.435 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.433+0000 7f2e22ffd700 1 -- 172.21.15.190:0/3375866678 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e28014420 con 0x7f2e340f47c0 2023-11-15T04:31:48.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.637+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e34107e90 con 0x7f2e340f47c0 2023-11-15T04:31:48.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.638+0000 7f2e22ffd700 1 -- 172.21.15.190:0/3375866678 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e2802b020 con 0x7f2e340f47c0 2023-11-15T04:31:48.639 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:48.640 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:48.640 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.639+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e1c02ff60 msgr2=0x7f2e1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:48.640 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.639+0000 7f2e38df5700 1 --2- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e1c02ff60 0x7f2e1c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2e240096a0 tx=0x7f2e24006b40).stop 2023-11-15T04:31:48.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.639+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 msgr2=0x7f2e34106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:48.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.639+0000 7f2e38df5700 1 --2- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 0x7f2e34106700 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2e28013510 tx=0x7f2e2800b040).stop 2023-11-15T04:31:48.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.639+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 shutdown_connections 2023-11-15T04:31:48.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.639+0000 7f2e38df5700 1 --2- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e1c02ff60 0x7f2e1c032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:48.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.639+0000 7f2e38df5700 1 --2- 172.21.15.190:0/3375866678 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e340f47c0 0x7f2e34106700 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:48.642 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.639+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 >> 172.21.15.190:0/3375866678 conn(0x7f2e340f02e0 msgr2=0x7f2e340f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:48.642 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.639+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 shutdown_connections 2023-11-15T04:31:48.642 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:48.640+0000 7f2e38df5700 1 -- 172.21.15.190:0/3375866678 wait complete. 2023-11-15T04:31:48.642 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:49.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:48 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:48.639563+0000 mon.smithi047 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.190:0/3375866678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:50.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:49 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:49.080914+0000 mgr.smithi047.zgysuz (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:50.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:50.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:51.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.150+0000 7f43a74f0700 1 -- 172.21.15.190:0/2645769110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 msgr2=0x7f43a00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:51.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.150+0000 7f43a74f0700 1 --2- 172.21.15.190:0/2645769110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 0x7f43a00f90f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f439c009fb0 tx=0x7f439c009b30).stop 2023-11-15T04:31:51.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.150+0000 7f43a74f0700 1 -- 172.21.15.190:0/2645769110 shutdown_connections 2023-11-15T04:31:51.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.150+0000 7f43a74f0700 1 --2- 172.21.15.190:0/2645769110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 0x7f43a00f90f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:51.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.150+0000 7f43a74f0700 1 -- 172.21.15.190:0/2645769110 >> 172.21.15.190:0/2645769110 conn(0x7f43a00f0b70 msgr2=0x7f43a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:51.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.150+0000 7f43a74f0700 1 -- 172.21.15.190:0/2645769110 shutdown_connections 2023-11-15T04:31:51.153 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.150+0000 7f43a74f0700 1 -- 172.21.15.190:0/2645769110 wait complete. 2023-11-15T04:31:51.154 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.151+0000 7f43a74f0700 1 Processor -- start 2023-11-15T04:31:51.154 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.151+0000 7f43a74f0700 1 -- start start 2023-11-15T04:31:51.155 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.151+0000 7f43a74f0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 0x7f43a01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:51.155 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.151+0000 7f43a74f0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a0106bd0 con 0x7f43a0060280 2023-11-15T04:31:51.155 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.151+0000 7f43a528c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 0x7f43a01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:51.155 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.151+0000 7f43a528c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 0x7f43a01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:55094/0 (socket says 172.21.15.190:55094) 2023-11-15T04:31:51.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.151+0000 7f43a528c700 1 -- 172.21.15.190:0/1033232715 learned_addr learned my addr 172.21.15.190:0/1033232715 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:51.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.152+0000 7f43a528c700 1 -- 172.21.15.190:0/1033232715 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c005040 con 0x7f43a0060280 2023-11-15T04:31:51.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.152+0000 7f43a528c700 1 --2- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 0x7f43a01066e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f439c009830 tx=0x7f439c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:51.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.152+0000 7f4395ffb700 1 -- 172.21.15.190:0/1033232715 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f439c009320 con 0x7f43a0060280 2023-11-15T04:31:51.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.152+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a0106dd0 con 0x7f43a0060280 2023-11-15T04:31:51.157 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.152+0000 7f4395ffb700 1 -- 172.21.15.190:0/1033232715 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f439c0136c0 con 0x7f43a0060280 2023-11-15T04:31:51.157 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.152+0000 7f4395ffb700 1 -- 172.21.15.190:0/1033232715 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f439c01ee90 con 0x7f43a0060280 2023-11-15T04:31:51.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.152+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a0107260 con 0x7f43a0060280 2023-11-15T04:31:51.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.153+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a0045ad0 con 0x7f43a0060280 2023-11-15T04:31:51.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.153+0000 7f4395ffb700 1 -- 172.21.15.190:0/1033232715 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f439c01e420 con 0x7f43a0060280 2023-11-15T04:31:51.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.153+0000 7f4395ffb700 1 --2- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f438c02ff10 0x7f438c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:51.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.153+0000 7f4395ffb700 1 -- 172.21.15.190:0/1033232715 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f439c040f70 con 0x7f43a0060280 2023-11-15T04:31:51.159 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.154+0000 7f43a4a8b700 1 --2- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f438c02ff10 0x7f438c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:51.163 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.154+0000 7f43a4a8b700 1 --2- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f438c02ff10 0x7f438c0323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f43900096a0 tx=0x7f4390006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:51.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.159+0000 7f4395ffb700 1 -- 172.21.15.190:0/1033232715 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f439c06d2b0 con 0x7f43a0060280 2023-11-15T04:31:51.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.364+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43a0108110 con 0x7f43a0060280 2023-11-15T04:31:51.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.365+0000 7f4395ffb700 1 -- 172.21.15.190:0/1033232715 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f439c02c2d0 con 0x7f43a0060280 2023-11-15T04:31:51.366 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:51.366 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:51.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f438c02ff10 msgr2=0x7f438c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:51.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 --2- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f438c02ff10 0x7f438c0323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f43900096a0 tx=0x7f4390006b40).stop 2023-11-15T04:31:51.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 msgr2=0x7f43a01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:51.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 --2- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 0x7f43a01066e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f439c009830 tx=0x7f439c00b040).stop 2023-11-15T04:31:51.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 shutdown_connections 2023-11-15T04:31:51.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 --2- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f438c02ff10 0x7f438c0323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:51.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 --2- 172.21.15.190:0/1033232715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43a0060280 0x7f43a01066e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:51.369 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 >> 172.21.15.190:0/1033232715 conn(0x7f43a00f0b70 msgr2=0x7f43a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:51.369 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 shutdown_connections 2023-11-15T04:31:51.369 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:51.366+0000 7f43a74f0700 1 -- 172.21.15.190:0/1033232715 wait complete. 2023-11-15T04:31:51.370 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:51.081228+0000 mgr.smithi047.zgysuz (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:51.366169+0000 mon.smithi047 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.190:0/1033232715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:52.773 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:52.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:53.921 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.920+0000 7f142ed20700 1 -- 172.21.15.190:0/1618077588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 msgr2=0x7f14280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:53.922 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.920+0000 7f142ed20700 1 --2- 172.21.15.190:0/1618077588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 0x7f14280f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f141c009fb0 tx=0x7f141c009b30).stop 2023-11-15T04:31:53.922 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.920+0000 7f142ed20700 1 -- 172.21.15.190:0/1618077588 shutdown_connections 2023-11-15T04:31:53.922 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.920+0000 7f142ed20700 1 --2- 172.21.15.190:0/1618077588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 0x7f14280f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:53.922 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.920+0000 7f142ed20700 1 -- 172.21.15.190:0/1618077588 >> 172.21.15.190:0/1618077588 conn(0x7f14280f0b70 msgr2=0x7f14280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:53.922 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.920+0000 7f142ed20700 1 -- 172.21.15.190:0/1618077588 shutdown_connections 2023-11-15T04:31:53.923 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.920+0000 7f142ed20700 1 -- 172.21.15.190:0/1618077588 wait complete. 2023-11-15T04:31:53.923 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.920+0000 7f142ed20700 1 Processor -- start 2023-11-15T04:31:53.923 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.920+0000 7f142ed20700 1 -- start start 2023-11-15T04:31:53.923 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.921+0000 7f142ed20700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 0x7f1428109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:53.923 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.921+0000 7f142ed20700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1428060220 con 0x7f14280f5410 2023-11-15T04:31:53.926 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.921+0000 7f142cabc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 0x7f1428109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:53.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.921+0000 7f142cabc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 0x7f1428109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:55106/0 (socket says 172.21.15.190:55106) 2023-11-15T04:31:53.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.921+0000 7f142cabc700 1 -- 172.21.15.190:0/4092338656 learned_addr learned my addr 172.21.15.190:0/4092338656 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:53.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.921+0000 7f142cabc700 1 -- 172.21.15.190:0/4092338656 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c005040 con 0x7f14280f5410 2023-11-15T04:31:53.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.921+0000 7f142cabc700 1 --2- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 0x7f1428109450 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f141c013510 tx=0x7f141c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:53.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.921+0000 7f14257fa700 1 -- 172.21.15.190:0/4092338656 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f141c009320 con 0x7f14280f5410 2023-11-15T04:31:53.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.921+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1428060420 con 0x7f14280f5410 2023-11-15T04:31:53.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.922+0000 7f14257fa700 1 -- 172.21.15.190:0/4092338656 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f141c013f00 con 0x7f14280f5410 2023-11-15T04:31:53.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.922+0000 7f14257fa700 1 -- 172.21.15.190:0/4092338656 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f141c014d90 con 0x7f14280f5410 2023-11-15T04:31:53.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.922+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1428109b00 con 0x7f14280f5410 2023-11-15T04:31:53.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.922+0000 7f14257fa700 1 -- 172.21.15.190:0/4092338656 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f141c014420 con 0x7f14280f5410 2023-11-15T04:31:53.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.922+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1428059100 con 0x7f14280f5410 2023-11-15T04:31:53.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.922+0000 7f14257fa700 1 --2- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f141802ff60 0x7f1418032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:53.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.923+0000 7f14257fa700 1 -- 172.21.15.190:0/4092338656 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f141c040ff0 con 0x7f14280f5410 2023-11-15T04:31:53.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.923+0000 7f1427fff700 1 --2- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f141802ff60 0x7f1418032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:53.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.923+0000 7f1427fff700 1 --2- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f141802ff60 0x7f1418032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f14140096a0 tx=0x7f1414006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:53.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:53.928+0000 7f14257fa700 1 -- 172.21.15.190:0/4092338656 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f141c0095f0 con 0x7f14280f5410 2023-11-15T04:31:54.137 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.136+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1428045ad0 con 0x7f14280f5410 2023-11-15T04:31:54.137 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.136+0000 7f14257fa700 1 -- 172.21.15.190:0/4092338656 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f141c01db70 con 0x7f14280f5410 2023-11-15T04:31:54.138 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:54.138 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:54.139 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f141802ff60 msgr2=0x7f1418032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:54.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 --2- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f141802ff60 0x7f1418032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f14140096a0 tx=0x7f1414006b40).stop 2023-11-15T04:31:54.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 msgr2=0x7f1428109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:54.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 --2- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 0x7f1428109450 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f141c013510 tx=0x7f141c00b040).stop 2023-11-15T04:31:54.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 shutdown_connections 2023-11-15T04:31:54.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 --2- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f141802ff60 0x7f1418032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:54.141 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 --2- 172.21.15.190:0/4092338656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14280f5410 0x7f1428109450 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:54.141 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 >> 172.21.15.190:0/4092338656 conn(0x7f14280f0b70 msgr2=0x7f14280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:54.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 shutdown_connections 2023-11-15T04:31:54.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:54.138+0000 7f142ed20700 1 -- 172.21.15.190:0/4092338656 wait complete. 2023-11-15T04:31:54.143 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:53.081571+0000 mgr.smithi047.zgysuz (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:55.505 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:55.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:55.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:55 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:54.138048+0000 mon.smithi047 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.190:0/4092338656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:55.081945+0000 mgr.smithi047.zgysuz (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:56.675 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.673+0000 7f3ca21dc700 1 -- 172.21.15.190:0/3867951205 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 msgr2=0x7f3c9c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:56.675 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.673+0000 7f3ca21dc700 1 --2- 172.21.15.190:0/3867951205 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 0x7f3c9c0f90f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f3c8c009fb0 tx=0x7f3c8c009b30).stop 2023-11-15T04:31:56.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.673+0000 7f3ca21dc700 1 -- 172.21.15.190:0/3867951205 shutdown_connections 2023-11-15T04:31:56.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.673+0000 7f3ca21dc700 1 --2- 172.21.15.190:0/3867951205 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 0x7f3c9c0f90f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:56.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.673+0000 7f3ca21dc700 1 -- 172.21.15.190:0/3867951205 >> 172.21.15.190:0/3867951205 conn(0x7f3c9c0f0b70 msgr2=0x7f3c9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:56.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.674+0000 7f3ca21dc700 1 -- 172.21.15.190:0/3867951205 shutdown_connections 2023-11-15T04:31:56.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.674+0000 7f3ca21dc700 1 -- 172.21.15.190:0/3867951205 wait complete. 2023-11-15T04:31:56.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.674+0000 7f3ca21dc700 1 Processor -- start 2023-11-15T04:31:56.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.674+0000 7f3ca21dc700 1 -- start start 2023-11-15T04:31:56.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.674+0000 7f3ca21dc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 0x7f3c9c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:56.678 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.674+0000 7f3ca21dc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c0f4ff0 con 0x7f3c9c060280 2023-11-15T04:31:56.678 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3c9b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 0x7f3c9c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:56.678 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3c9b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 0x7f3c9c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:55118/0 (socket says 172.21.15.190:55118) 2023-11-15T04:31:56.678 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3c9b7fe700 1 -- 172.21.15.190:0/953159417 learned_addr learned my addr 172.21.15.190:0/953159417 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:56.679 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3c9b7fe700 1 -- 172.21.15.190:0/953159417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c005040 con 0x7f3c9c060280 2023-11-15T04:31:56.679 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3c9b7fe700 1 --2- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 0x7f3c9c0f6b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f3c8c013510 tx=0x7f3c8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:56.679 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3c93fff700 1 -- 172.21.15.190:0/953159417 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c8c009320 con 0x7f3c9c060280 2023-11-15T04:31:56.679 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c0f5190 con 0x7f3c9c060280 2023-11-15T04:31:56.679 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3c93fff700 1 -- 172.21.15.190:0/953159417 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c8c013f00 con 0x7f3c9c060280 2023-11-15T04:31:56.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3c93fff700 1 -- 172.21.15.190:0/953159417 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c8c014dc0 con 0x7f3c9c060280 2023-11-15T04:31:56.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.675+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9c0f5620 con 0x7f3c9c060280 2023-11-15T04:31:56.681 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.676+0000 7f3c93fff700 1 -- 172.21.15.190:0/953159417 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c8c014420 con 0x7f3c9c060280 2023-11-15T04:31:56.681 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.676+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c059100 con 0x7f3c9c060280 2023-11-15T04:31:56.681 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.676+0000 7f3c93fff700 1 --2- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3c8802ff60 0x7f3c88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:56.681 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.676+0000 7f3c93fff700 1 -- 172.21.15.190:0/953159417 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c8c041020 con 0x7f3c9c060280 2023-11-15T04:31:56.682 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.676+0000 7f3c9affd700 1 --2- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3c8802ff60 0x7f3c88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:56.682 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.677+0000 7f3c9affd700 1 --2- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3c8802ff60 0x7f3c88032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f3c840096a0 tx=0x7f3c84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:56.682 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.681+0000 7f3c93fff700 1 -- 172.21.15.190:0/953159417 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c8c0095f0 con 0x7f3c9c060280 2023-11-15T04:31:56.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.889+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c9c045ad0 con 0x7f3c9c060280 2023-11-15T04:31:56.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.890+0000 7f3c93fff700 1 -- 172.21.15.190:0/953159417 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c8c01cb10 con 0x7f3c9c060280 2023-11-15T04:31:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:56.890 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:56.892 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3c8802ff60 msgr2=0x7f3c88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:56.892 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 --2- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3c8802ff60 0x7f3c88032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f3c840096a0 tx=0x7f3c84006b40).stop 2023-11-15T04:31:56.892 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 msgr2=0x7f3c9c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:56.893 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 --2- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 0x7f3c9c0f6b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f3c8c013510 tx=0x7f3c8c00b040).stop 2023-11-15T04:31:56.893 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 shutdown_connections 2023-11-15T04:31:56.893 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 --2- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3c8802ff60 0x7f3c88032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:56.893 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 --2- 172.21.15.190:0/953159417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c9c060280 0x7f3c9c0f6b70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:56.894 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 >> 172.21.15.190:0/953159417 conn(0x7f3c9c0f0b70 msgr2=0x7f3c9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:56.894 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 shutdown_connections 2023-11-15T04:31:56.894 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:56.891+0000 7f3ca21dc700 1 -- 172.21.15.190:0/953159417 wait complete. 2023-11-15T04:31:56.894 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:31:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:56.890193+0000 mon.smithi047 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.190:0/953159417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:31:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:31:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:57.082356+0000 mgr.smithi047.zgysuz (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:31:58.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:31:58.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:31:59.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.389+0000 7fc6e202c700 1 -- 172.21.15.190:0/2511687150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 msgr2=0x7fc6dc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:59.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.389+0000 7fc6e202c700 1 --2- 172.21.15.190:0/2511687150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 0x7fc6dc0f66c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fc6cc009fb0 tx=0x7fc6cc009b30).stop 2023-11-15T04:31:59.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.390+0000 7fc6e202c700 1 -- 172.21.15.190:0/2511687150 shutdown_connections 2023-11-15T04:31:59.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.390+0000 7fc6e202c700 1 --2- 172.21.15.190:0/2511687150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 0x7fc6dc0f66c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:59.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.390+0000 7fc6e202c700 1 -- 172.21.15.190:0/2511687150 >> 172.21.15.190:0/2511687150 conn(0x7fc6dc0f1aa0 msgr2=0x7fc6dc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:59.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.390+0000 7fc6e202c700 1 -- 172.21.15.190:0/2511687150 shutdown_connections 2023-11-15T04:31:59.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.390+0000 7fc6e202c700 1 -- 172.21.15.190:0/2511687150 wait complete. 2023-11-15T04:31:59.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.390+0000 7fc6e202c700 1 Processor -- start 2023-11-15T04:31:59.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.390+0000 7fc6e202c700 1 -- start start 2023-11-15T04:31:59.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.390+0000 7fc6e202c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 0x7fc6dc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:59.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.390+0000 7fc6e202c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc069e20 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6e102a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 0x7fc6dc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:59.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6e102a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 0x7fc6dc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:55132/0 (socket says 172.21.15.190:55132) 2023-11-15T04:31:59.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6e102a700 1 -- 172.21.15.190:0/179551565 learned_addr learned my addr 172.21.15.190:0/179551565 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:31:59.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6e102a700 1 -- 172.21.15.190:0/179551565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc005040 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6e102a700 1 --2- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 0x7fc6dc06b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fc6cc013510 tx=0x7fc6cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:59.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6d1ffb700 1 -- 172.21.15.190:0/179551565 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6cc009320 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc069fc0 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6d1ffb700 1 -- 172.21.15.190:0/179551565 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6cc013f00 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6d1ffb700 1 -- 172.21.15.190:0/179551565 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6cc014dc0 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.391+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc06a450 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.392+0000 7fc6d1ffb700 1 -- 172.21.15.190:0/179551565 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6cc0095f0 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.392+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc059100 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.392+0000 7fc6d1ffb700 1 --2- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc6c802ff60 0x7fc6c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:31:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.392+0000 7fc6d1ffb700 1 -- 172.21.15.190:0/179551565 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6cc041100 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.392+0000 7fc6e0829700 1 --2- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc6c802ff60 0x7fc6c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:31:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.393+0000 7fc6e0829700 1 --2- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc6c802ff60 0x7fc6c8032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fc6d80096a0 tx=0x7fc6d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:31:59.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.397+0000 7fc6d1ffb700 1 -- 172.21.15.190:0/179551565 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6cc014420 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.602+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6dc0fbce0 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.603+0000 7fc6d1ffb700 1 -- 172.21.15.190:0/179551565 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc6cc010230 con 0x7fc6dc0f62a0 2023-11-15T04:31:59.603 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:31:59.604 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:31:59.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc6c802ff60 msgr2=0x7fc6c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:59.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 --2- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc6c802ff60 0x7fc6c8032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fc6d80096a0 tx=0x7fc6d8006b40).stop 2023-11-15T04:31:59.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 msgr2=0x7fc6dc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:31:59.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 --2- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 0x7fc6dc06b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fc6cc013510 tx=0x7fc6cc00b040).stop 2023-11-15T04:31:59.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 shutdown_connections 2023-11-15T04:31:59.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 --2- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc6c802ff60 0x7fc6c8032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:59.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 --2- 172.21.15.190:0/179551565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6dc0f62a0 0x7fc6dc06b9a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:31:59.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 >> 172.21.15.190:0/179551565 conn(0x7fc6dc0f1aa0 msgr2=0x7fc6dc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:31:59.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 shutdown_connections 2023-11-15T04:31:59.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:31:59.605+0000 7fc6e202c700 1 -- 172.21.15.190:0/179551565 wait complete. 2023-11-15T04:31:59.607 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:31:59.082760+0000 mgr.smithi047.zgysuz (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:31:59.603660+0000 mon.smithi047 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.190:0/179551565' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:00.975 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:00.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:02.143 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.143+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/3040877017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 msgr2=0x7f4f680f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:02.144 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.143+0000 7f4f6fcf6700 1 --2- 172.21.15.190:0/3040877017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 0x7f4f680f90f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f4f58009fb0 tx=0x7f4f58009b30).stop 2023-11-15T04:32:02.144 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.143+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/3040877017 shutdown_connections 2023-11-15T04:32:02.144 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.143+0000 7f4f6fcf6700 1 --2- 172.21.15.190:0/3040877017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 0x7f4f680f90f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:02.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.143+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/3040877017 >> 172.21.15.190:0/3040877017 conn(0x7f4f680f0b70 msgr2=0x7f4f680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:02.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.143+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/3040877017 shutdown_connections 2023-11-15T04:32:02.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.143+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/3040877017 wait complete. 2023-11-15T04:32:02.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.144+0000 7f4f6fcf6700 1 Processor -- start 2023-11-15T04:32:02.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.144+0000 7f4f6fcf6700 1 -- start start 2023-11-15T04:32:02.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.144+0000 7f4f6fcf6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 0x7f4f681088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:02.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.144+0000 7f4f6fcf6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68108dc0 con 0x7f4f68060280 2023-11-15T04:32:02.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.144+0000 7f4f6da92700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 0x7f4f681088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:02.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.144+0000 7f4f6da92700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 0x7f4f681088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60122/0 (socket says 172.21.15.190:60122) 2023-11-15T04:32:02.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.144+0000 7f4f6da92700 1 -- 172.21.15.190:0/4120893495 learned_addr learned my addr 172.21.15.190:0/4120893495 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:02.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.144+0000 7f4f6da92700 1 -- 172.21.15.190:0/4120893495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f58005040 con 0x7f4f68060280 2023-11-15T04:32:02.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.145+0000 7f4f6da92700 1 --2- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 0x7f4f681088d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4f58013510 tx=0x7f4f5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:02.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.145+0000 7f4f5e7fc700 1 -- 172.21.15.190:0/4120893495 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f58009320 con 0x7f4f68060280 2023-11-15T04:32:02.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.145+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f68108fc0 con 0x7f4f68060280 2023-11-15T04:32:02.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.145+0000 7f4f5e7fc700 1 -- 172.21.15.190:0/4120893495 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f58013f00 con 0x7f4f68060280 2023-11-15T04:32:02.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.145+0000 7f4f5e7fc700 1 -- 172.21.15.190:0/4120893495 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f58014dc0 con 0x7f4f68060280 2023-11-15T04:32:02.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.145+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f68109450 con 0x7f4f68060280 2023-11-15T04:32:02.150 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.146+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f68045ad0 con 0x7f4f68060280 2023-11-15T04:32:02.150 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.146+0000 7f4f5e7fc700 1 -- 172.21.15.190:0/4120893495 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f580095f0 con 0x7f4f68060280 2023-11-15T04:32:02.150 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.146+0000 7f4f5e7fc700 1 --2- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4f5402ff60 0x7f4f54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:02.150 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.146+0000 7f4f5e7fc700 1 -- 172.21.15.190:0/4120893495 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f58041040 con 0x7f4f68060280 2023-11-15T04:32:02.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.146+0000 7f4f6d291700 1 --2- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4f5402ff60 0x7f4f54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:02.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.146+0000 7f4f6d291700 1 --2- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4f5402ff60 0x7f4f54032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4f640096a0 tx=0x7f4f64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:02.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.151+0000 7f4f5e7fc700 1 -- 172.21.15.190:0/4120893495 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f58014420 con 0x7f4f68060280 2023-11-15T04:32:02.356 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.356+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f6810a300 con 0x7f4f68060280 2023-11-15T04:32:02.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.357+0000 7f4f5e7fc700 1 -- 172.21.15.190:0/4120893495 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f58010230 con 0x7f4f68060280 2023-11-15T04:32:02.358 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:02.358 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:02.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4f5402ff60 msgr2=0x7f4f54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:02.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 --2- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4f5402ff60 0x7f4f54032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4f640096a0 tx=0x7f4f64006b40).stop 2023-11-15T04:32:02.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 msgr2=0x7f4f681088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:02.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 --2- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 0x7f4f681088d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4f58013510 tx=0x7f4f5800b040).stop 2023-11-15T04:32:02.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 shutdown_connections 2023-11-15T04:32:02.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 --2- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4f5402ff60 0x7f4f54032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:02.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 --2- 172.21.15.190:0/4120893495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f68060280 0x7f4f681088d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:02.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 >> 172.21.15.190:0/4120893495 conn(0x7f4f680f0b70 msgr2=0x7f4f680f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:02.361 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 shutdown_connections 2023-11-15T04:32:02.361 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:02.359+0000 7f4f6fcf6700 1 -- 172.21.15.190:0/4120893495 wait complete. 2023-11-15T04:32:02.361 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:01.083032+0000 mgr.smithi047.zgysuz (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:03.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:03 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:02.357715+0000 mon.smithi047 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.190:0/4120893495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:03.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:03.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:03.083314+0000 mgr.smithi047.zgysuz (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:04.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.880+0000 7fd681049700 1 -- 172.21.15.190:0/1119346605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 msgr2=0x7fd67c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:04.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.880+0000 7fd681049700 1 --2- 172.21.15.190:0/1119346605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 0x7fd67c0f6930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fd66c009fb0 tx=0x7fd66c009b30).stop 2023-11-15T04:32:04.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.880+0000 7fd681049700 1 -- 172.21.15.190:0/1119346605 shutdown_connections 2023-11-15T04:32:04.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.880+0000 7fd681049700 1 --2- 172.21.15.190:0/1119346605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 0x7fd67c0f6930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:04.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.880+0000 7fd681049700 1 -- 172.21.15.190:0/1119346605 >> 172.21.15.190:0/1119346605 conn(0x7fd67c0f1c70 msgr2=0x7fd67c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:04.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.880+0000 7fd681049700 1 -- 172.21.15.190:0/1119346605 shutdown_connections 2023-11-15T04:32:04.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.880+0000 7fd681049700 1 -- 172.21.15.190:0/1119346605 wait complete. 2023-11-15T04:32:04.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.881+0000 7fd681049700 1 Processor -- start 2023-11-15T04:32:04.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.881+0000 7fd681049700 1 -- start start 2023-11-15T04:32:04.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.881+0000 7fd681049700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 0x7fd67c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:04.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.881+0000 7fd681049700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c106c20 con 0x7fd67c0f6510 2023-11-15T04:32:04.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.881+0000 7fd67ad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 0x7fd67c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:04.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.881+0000 7fd67ad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 0x7fd67c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60150/0 (socket says 172.21.15.190:60150) 2023-11-15T04:32:04.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.881+0000 7fd67ad9d700 1 -- 172.21.15.190:0/3452575922 learned_addr learned my addr 172.21.15.190:0/3452575922 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:04.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.882+0000 7fd67ad9d700 1 -- 172.21.15.190:0/3452575922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c005040 con 0x7fd67c0f6510 2023-11-15T04:32:04.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.882+0000 7fd67ad9d700 1 --2- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 0x7fd67c106730 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd66c013510 tx=0x7fd66c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:04.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.882+0000 7fd6637fe700 1 -- 172.21.15.190:0/3452575922 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66c009320 con 0x7fd67c0f6510 2023-11-15T04:32:04.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.882+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67c106e20 con 0x7fd67c0f6510 2023-11-15T04:32:04.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.882+0000 7fd6637fe700 1 -- 172.21.15.190:0/3452575922 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd66c013f00 con 0x7fd67c0f6510 2023-11-15T04:32:04.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.882+0000 7fd6637fe700 1 -- 172.21.15.190:0/3452575922 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66c014d90 con 0x7fd67c0f6510 2023-11-15T04:32:04.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.882+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67c1072b0 con 0x7fd67c0f6510 2023-11-15T04:32:04.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.883+0000 7fd6637fe700 1 -- 172.21.15.190:0/3452575922 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd66c0095f0 con 0x7fd67c0f6510 2023-11-15T04:32:04.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.883+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67c045ad0 con 0x7fd67c0f6510 2023-11-15T04:32:04.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.883+0000 7fd6637fe700 1 --2- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd66402ff60 0x7fd664032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:04.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.883+0000 7fd6637fe700 1 -- 172.21.15.190:0/3452575922 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd66c040fe0 con 0x7fd67c0f6510 2023-11-15T04:32:04.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.883+0000 7fd67a59c700 1 --2- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd66402ff60 0x7fd664032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:04.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.884+0000 7fd67a59c700 1 --2- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd66402ff60 0x7fd664032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd6700096a0 tx=0x7fd670006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:04.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:04.888+0000 7fd6637fe700 1 -- 172.21.15.190:0/3452575922 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd66c01d7f0 con 0x7fd67c0f6510 2023-11-15T04:32:05.092 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.092+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd67c0fbf60 con 0x7fd67c0f6510 2023-11-15T04:32:05.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.093+0000 7fd6637fe700 1 -- 172.21.15.190:0/3452575922 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd66c010230 con 0x7fd67c0f6510 2023-11-15T04:32:05.094 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:05.094 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:05.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd66402ff60 msgr2=0x7fd664032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:05.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 --2- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd66402ff60 0x7fd664032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd6700096a0 tx=0x7fd670006b40).stop 2023-11-15T04:32:05.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 msgr2=0x7fd67c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:05.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 --2- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 0x7fd67c106730 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd66c013510 tx=0x7fd66c00b040).stop 2023-11-15T04:32:05.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 shutdown_connections 2023-11-15T04:32:05.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 --2- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd66402ff60 0x7fd664032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:05.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 --2- 172.21.15.190:0/3452575922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd67c0f6510 0x7fd67c106730 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:05.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 >> 172.21.15.190:0/3452575922 conn(0x7fd67c0f1c70 msgr2=0x7fd67c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:05.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 shutdown_connections 2023-11-15T04:32:05.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:05.095+0000 7fd681049700 1 -- 172.21.15.190:0/3452575922 wait complete. 2023-11-15T04:32:05.097 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:05.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:05 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:05.093765+0000 mon.smithi047 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.190:0/3452575922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:06.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:06.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:05.083576+0000 mgr.smithi047.zgysuz (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:06.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:05.216153+0000 mon.smithi047 (mon.0) 177 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:32:06.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:05.218671+0000 mon.smithi047 (mon.0) 178 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:32:07.691 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.691+0000 7f98a3529700 1 -- 172.21.15.190:0/2081097632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 msgr2=0x7f989c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:07.692 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.691+0000 7f98a3529700 1 --2- 172.21.15.190:0/2081097632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 0x7f989c0f9190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f9898009fb0 tx=0x7f9898009b30).stop 2023-11-15T04:32:07.692 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.691+0000 7f98a3529700 1 -- 172.21.15.190:0/2081097632 shutdown_connections 2023-11-15T04:32:07.692 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.691+0000 7f98a3529700 1 --2- 172.21.15.190:0/2081097632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 0x7f989c0f9190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:07.692 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.691+0000 7f98a3529700 1 -- 172.21.15.190:0/2081097632 >> 172.21.15.190:0/2081097632 conn(0x7f989c0f0b70 msgr2=0x7f989c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:07.693 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.691+0000 7f98a3529700 1 -- 172.21.15.190:0/2081097632 shutdown_connections 2023-11-15T04:32:07.693 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.691+0000 7f98a3529700 1 -- 172.21.15.190:0/2081097632 wait complete. 2023-11-15T04:32:07.693 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.692+0000 7f98a3529700 1 Processor -- start 2023-11-15T04:32:07.693 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.692+0000 7f98a3529700 1 -- start start 2023-11-15T04:32:07.693 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.692+0000 7f98a3529700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 0x7f989c108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:07.694 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.692+0000 7f98a3529700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c108df0 con 0x7f989c0601e0 2023-11-15T04:32:07.695 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.692+0000 7f98a12c5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 0x7f989c108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:07.696 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.692+0000 7f98a12c5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 0x7f989c108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60166/0 (socket says 172.21.15.190:60166) 2023-11-15T04:32:07.696 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.692+0000 7f98a12c5700 1 -- 172.21.15.190:0/2653151725 learned_addr learned my addr 172.21.15.190:0/2653151725 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:07.696 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.692+0000 7f98a12c5700 1 -- 172.21.15.190:0/2653151725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9898005040 con 0x7f989c0601e0 2023-11-15T04:32:07.696 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.692+0000 7f98a12c5700 1 --2- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 0x7f989c108900 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f9898004dd0 tx=0x7f9898009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:07.697 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.693+0000 7f9891ffb700 1 -- 172.21.15.190:0/2653151725 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9898018070 con 0x7f989c0601e0 2023-11-15T04:32:07.697 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.693+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989c108ff0 con 0x7f989c0601e0 2023-11-15T04:32:07.697 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.693+0000 7f9891ffb700 1 -- 172.21.15.190:0/2653151725 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9898013410 con 0x7f989c0601e0 2023-11-15T04:32:07.697 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.693+0000 7f9891ffb700 1 -- 172.21.15.190:0/2653151725 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f989801c3c0 con 0x7f989c0601e0 2023-11-15T04:32:07.698 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.693+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989c109480 con 0x7f989c0601e0 2023-11-15T04:32:07.698 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.694+0000 7f9891ffb700 1 -- 172.21.15.190:0/2653151725 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f989801d420 con 0x7f989c0601e0 2023-11-15T04:32:07.698 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.694+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f989c045ad0 con 0x7f989c0601e0 2023-11-15T04:32:07.698 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.694+0000 7f9891ffb700 1 --2- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f988802ff60 0x7f9888032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:07.698 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.694+0000 7f9891ffb700 1 -- 172.21.15.190:0/2653151725 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9898041450 con 0x7f989c0601e0 2023-11-15T04:32:07.699 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.694+0000 7f98a0ac4700 1 --2- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f988802ff60 0x7f9888032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:07.699 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.695+0000 7f98a0ac4700 1 --2- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f988802ff60 0x7f9888032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f988c0096a0 tx=0x7f988c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:07.699 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.699+0000 7f9891ffb700 1 -- 172.21.15.190:0/2653151725 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f989801d820 con 0x7f989c0601e0 2023-11-15T04:32:07.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.903+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f989c10a330 con 0x7f989c0601e0 2023-11-15T04:32:07.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.904+0000 7f9891ffb700 1 -- 172.21.15.190:0/2653151725 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f989801b370 con 0x7f989c0601e0 2023-11-15T04:32:07.904 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:07.905 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:07.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f988802ff60 msgr2=0x7f9888032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:07.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 --2- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f988802ff60 0x7f9888032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f988c0096a0 tx=0x7f988c006b40).stop 2023-11-15T04:32:07.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 msgr2=0x7f989c108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:07.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 --2- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 0x7f989c108900 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f9898004dd0 tx=0x7f9898009b30).stop 2023-11-15T04:32:07.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 shutdown_connections 2023-11-15T04:32:07.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 --2- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f988802ff60 0x7f9888032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:07.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 --2- 172.21.15.190:0/2653151725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f989c0601e0 0x7f989c108900 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:07.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 >> 172.21.15.190:0/2653151725 conn(0x7f989c0f0b70 msgr2=0x7f989c0f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:07.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 shutdown_connections 2023-11-15T04:32:07.908 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:07.906+0000 7f98a3529700 1 -- 172.21.15.190:0/2653151725 wait complete. 2023-11-15T04:32:07.908 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:08.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:07.083847+0000 mgr.smithi047.zgysuz (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:09.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:07.904665+0000 mon.smithi047 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.190:0/2653151725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:09.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:09.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:10.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:09.084189+0000 mgr.smithi047.zgysuz (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:10.388 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.387+0000 7fefeb494700 1 -- 172.21.15.190:0/2691010298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 msgr2=0x7fefe40f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:10.388 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.387+0000 7fefeb494700 1 --2- 172.21.15.190:0/2691010298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 0x7fefe40f6e20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fefd4009fb0 tx=0x7fefd4009b30).stop 2023-11-15T04:32:10.389 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.387+0000 7fefeb494700 1 -- 172.21.15.190:0/2691010298 shutdown_connections 2023-11-15T04:32:10.389 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.387+0000 7fefeb494700 1 --2- 172.21.15.190:0/2691010298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 0x7fefe40f6e20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:10.389 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.387+0000 7fefeb494700 1 -- 172.21.15.190:0/2691010298 >> 172.21.15.190:0/2691010298 conn(0x7fefe40f0260 msgr2=0x7fefe40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:10.389 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.387+0000 7fefeb494700 1 -- 172.21.15.190:0/2691010298 shutdown_connections 2023-11-15T04:32:10.389 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.387+0000 7fefeb494700 1 -- 172.21.15.190:0/2691010298 wait complete. 2023-11-15T04:32:10.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.388+0000 7fefeb494700 1 Processor -- start 2023-11-15T04:32:10.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.388+0000 7fefeb494700 1 -- start start 2023-11-15T04:32:10.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.388+0000 7fefeb494700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 0x7fefe41093b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:10.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.388+0000 7fefeb494700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe40f52b0 con 0x7fefe40f6a00 2023-11-15T04:32:10.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.388+0000 7fefe9230700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 0x7fefe41093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:10.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.388+0000 7fefe9230700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 0x7fefe41093b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56202/0 (socket says 172.21.15.190:56202) 2023-11-15T04:32:10.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.388+0000 7fefe9230700 1 -- 172.21.15.190:0/4211655935 learned_addr learned my addr 172.21.15.190:0/4211655935 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:10.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.388+0000 7fefe9230700 1 -- 172.21.15.190:0/4211655935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd4005040 con 0x7fefe40f6a00 2023-11-15T04:32:10.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.389+0000 7fefe9230700 1 --2- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 0x7fefe41093b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fefd4013510 tx=0x7fefd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:10.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.389+0000 7fefd9ffb700 1 -- 172.21.15.190:0/4211655935 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefd4009320 con 0x7fefe40f6a00 2023-11-15T04:32:10.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.389+0000 7fefd9ffb700 1 -- 172.21.15.190:0/4211655935 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefd4013f00 con 0x7fefe40f6a00 2023-11-15T04:32:10.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.389+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe40f54b0 con 0x7fefe40f6a00 2023-11-15T04:32:10.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.389+0000 7fefd9ffb700 1 -- 172.21.15.190:0/4211655935 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefd4014dc0 con 0x7fefe40f6a00 2023-11-15T04:32:10.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.389+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe4109a60 con 0x7fefe40f6a00 2023-11-15T04:32:10.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.390+0000 7fefd9ffb700 1 -- 172.21.15.190:0/4211655935 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefd40095f0 con 0x7fefe40f6a00 2023-11-15T04:32:10.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.390+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe4045ad0 con 0x7fefe40f6a00 2023-11-15T04:32:10.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.390+0000 7fefd9ffb700 1 --2- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fefd002ff60 0x7fefd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:10.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.390+0000 7fefd9ffb700 1 -- 172.21.15.190:0/4211655935 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefd4041100 con 0x7fefe40f6a00 2023-11-15T04:32:10.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.390+0000 7fefe8a2f700 1 --2- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fefd002ff60 0x7fefd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:10.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.391+0000 7fefe8a2f700 1 --2- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fefd002ff60 0x7fefd0032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fefe00096a0 tx=0x7fefe0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:10.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.395+0000 7fefd9ffb700 1 -- 172.21.15.190:0/4211655935 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefd4014420 con 0x7fefe40f6a00 2023-11-15T04:32:10.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.600+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefe40faef0 con 0x7fefe40f6a00 2023-11-15T04:32:10.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.601+0000 7fefd9ffb700 1 -- 172.21.15.190:0/4211655935 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fefd4010230 con 0x7fefe40f6a00 2023-11-15T04:32:10.601 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:10.601 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:10.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.602+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fefd002ff60 msgr2=0x7fefd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:10.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.602+0000 7fefeb494700 1 --2- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fefd002ff60 0x7fefd0032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fefe00096a0 tx=0x7fefe0006b40).stop 2023-11-15T04:32:10.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.602+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 msgr2=0x7fefe41093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:10.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.602+0000 7fefeb494700 1 --2- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 0x7fefe41093b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fefd4013510 tx=0x7fefd400b040).stop 2023-11-15T04:32:10.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.603+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 shutdown_connections 2023-11-15T04:32:10.603 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.603+0000 7fefeb494700 1 --2- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fefd002ff60 0x7fefd0032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:10.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.603+0000 7fefeb494700 1 --2- 172.21.15.190:0/4211655935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefe40f6a00 0x7fefe41093b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:10.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.603+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 >> 172.21.15.190:0/4211655935 conn(0x7fefe40f0260 msgr2=0x7fefe40f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:10.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.603+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 shutdown_connections 2023-11-15T04:32:10.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:10.603+0000 7fefeb494700 1 -- 172.21.15.190:0/4211655935 wait complete. 2023-11-15T04:32:10.605 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:11.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:10.601430+0000 mon.smithi047 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.190:0/4211655935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:11.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:11.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:12.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:11.084459+0000 mgr.smithi047.zgysuz (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:13.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.114+0000 7fbea0d14700 1 -- 172.21.15.190:0/1920841555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 msgr2=0x7fbe9c0f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:13.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.114+0000 7fbea0d14700 1 --2- 172.21.15.190:0/1920841555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 0x7fbe9c0f6e20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fbe90009fb0 tx=0x7fbe90009b30).stop 2023-11-15T04:32:13.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.114+0000 7fbea0d14700 1 -- 172.21.15.190:0/1920841555 shutdown_connections 2023-11-15T04:32:13.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.114+0000 7fbea0d14700 1 --2- 172.21.15.190:0/1920841555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 0x7fbe9c0f6e20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:13.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.114+0000 7fbea0d14700 1 -- 172.21.15.190:0/1920841555 >> 172.21.15.190:0/1920841555 conn(0x7fbe9c0f0260 msgr2=0x7fbe9c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:13.116 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.115+0000 7fbea0d14700 1 -- 172.21.15.190:0/1920841555 shutdown_connections 2023-11-15T04:32:13.116 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.115+0000 7fbea0d14700 1 -- 172.21.15.190:0/1920841555 wait complete. 2023-11-15T04:32:13.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.115+0000 7fbea0d14700 1 Processor -- start 2023-11-15T04:32:13.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.115+0000 7fbea0d14700 1 -- start start 2023-11-15T04:32:13.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.115+0000 7fbea0d14700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 0x7fbe9c106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:13.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.115+0000 7fbea0d14700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c106b80 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.115+0000 7fbe9a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 0x7fbe9c106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.115+0000 7fbe9a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 0x7fbe9c106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56232/0 (socket says 172.21.15.190:56232) 2023-11-15T04:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.115+0000 7fbe9a59c700 1 -- 172.21.15.190:0/3736031200 learned_addr learned my addr 172.21.15.190:0/3736031200 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.116+0000 7fbe9a59c700 1 -- 172.21.15.190:0/3736031200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe90005040 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.116+0000 7fbe9a59c700 1 --2- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 0x7fbe9c106690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fbe90009830 tx=0x7fbe9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.116+0000 7fbe8affd700 1 -- 172.21.15.190:0/3736031200 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe90009320 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.116+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c106d80 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.116+0000 7fbe8affd700 1 -- 172.21.15.190:0/3736031200 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe900136c0 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.116+0000 7fbe8affd700 1 -- 172.21.15.190:0/3736031200 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9001ee90 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.116+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c107210 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.117+0000 7fbe8affd700 1 -- 172.21.15.190:0/3736031200 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe9001e420 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.117+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c045ad0 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.117+0000 7fbe8affd700 1 --2- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe8402ff10 0x7fbe840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.117+0000 7fbe8affd700 1 -- 172.21.15.190:0/3736031200 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe90040f30 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.117+0000 7fbe99d9b700 1 --2- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe8402ff10 0x7fbe840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:13.122 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.118+0000 7fbe99d9b700 1 --2- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe8402ff10 0x7fbe840323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fbe8c0096a0 tx=0x7fbe8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:13.122 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.122+0000 7fbe8affd700 1 -- 172.21.15.190:0/3736031200 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe9006d2b0 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.330 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.330+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe9c1080c0 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.331 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.331+0000 7fbe8affd700 1 -- 172.21.15.190:0/3736031200 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe9001c790 con 0x7fbe9c0f6a00 2023-11-15T04:32:13.331 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:13.332 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:13.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe8402ff10 msgr2=0x7fbe840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:13.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 --2- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe8402ff10 0x7fbe840323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fbe8c0096a0 tx=0x7fbe8c006b40).stop 2023-11-15T04:32:13.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 msgr2=0x7fbe9c106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:13.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 --2- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 0x7fbe9c106690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fbe90009830 tx=0x7fbe9000b040).stop 2023-11-15T04:32:13.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 shutdown_connections 2023-11-15T04:32:13.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 --2- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe8402ff10 0x7fbe840323d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:13.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 --2- 172.21.15.190:0/3736031200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe9c0f6a00 0x7fbe9c106690 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:13.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 >> 172.21.15.190:0/3736031200 conn(0x7fbe9c0f0260 msgr2=0x7fbe9c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:13.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 shutdown_connections 2023-11-15T04:32:13.335 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:13.333+0000 7fbea0d14700 1 -- 172.21.15.190:0/3736031200 wait complete. 2023-11-15T04:32:13.335 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:13.084767+0000 mgr.smithi047.zgysuz (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:13.331504+0000 mon.smithi047 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.190:0/3736031200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:14.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:14.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:15.778 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.778+0000 7f05c05a9700 1 -- 172.21.15.190:0/457004026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 msgr2=0x7f05b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.778+0000 7f05c05a9700 1 --2- 172.21.15.190:0/457004026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 0x7f05b80f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f05ac009fb0 tx=0x7f05ac009b30).stop 2023-11-15T04:32:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.778+0000 7f05c05a9700 1 -- 172.21.15.190:0/457004026 shutdown_connections 2023-11-15T04:32:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.778+0000 7f05c05a9700 1 --2- 172.21.15.190:0/457004026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 0x7f05b80f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.778+0000 7f05c05a9700 1 -- 172.21.15.190:0/457004026 >> 172.21.15.190:0/457004026 conn(0x7f05b80f0b70 msgr2=0x7f05b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.778+0000 7f05c05a9700 1 -- 172.21.15.190:0/457004026 shutdown_connections 2023-11-15T04:32:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.778+0000 7f05c05a9700 1 -- 172.21.15.190:0/457004026 wait complete. 2023-11-15T04:32:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.779+0000 7f05c05a9700 1 Processor -- start 2023-11-15T04:32:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.779+0000 7f05c05a9700 1 -- start start 2023-11-15T04:32:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.779+0000 7f05c05a9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 0x7f05b8108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:15.781 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.779+0000 7f05c05a9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b8108e00 con 0x7f05b80f5410 2023-11-15T04:32:15.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.779+0000 7f05be345700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 0x7f05b8108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:15.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.779+0000 7f05be345700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 0x7f05b8108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56242/0 (socket says 172.21.15.190:56242) 2023-11-15T04:32:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.779+0000 7f05be345700 1 -- 172.21.15.190:0/1190891404 learned_addr learned my addr 172.21.15.190:0/1190891404 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.780+0000 7f05be345700 1 -- 172.21.15.190:0/1190891404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac005040 con 0x7f05b80f5410 2023-11-15T04:32:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.780+0000 7f05be345700 1 --2- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 0x7f05b8108910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f05ac013510 tx=0x7f05ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.780+0000 7f05aaffd700 1 -- 172.21.15.190:0/1190891404 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05ac009320 con 0x7f05b80f5410 2023-11-15T04:32:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.780+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b8109000 con 0x7f05b80f5410 2023-11-15T04:32:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.780+0000 7f05aaffd700 1 -- 172.21.15.190:0/1190891404 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05ac013f00 con 0x7f05b80f5410 2023-11-15T04:32:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.780+0000 7f05aaffd700 1 -- 172.21.15.190:0/1190891404 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05ac014d90 con 0x7f05b80f5410 2023-11-15T04:32:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.780+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b8109490 con 0x7f05b80f5410 2023-11-15T04:32:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.781+0000 7f05aaffd700 1 -- 172.21.15.190:0/1190891404 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05ac0263a0 con 0x7f05b80f5410 2023-11-15T04:32:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.781+0000 7f05aaffd700 1 --2- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f05a402ff60 0x7f05a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:15.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.781+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b8045ad0 con 0x7f05b80f5410 2023-11-15T04:32:15.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.781+0000 7f05aaffd700 1 -- 172.21.15.190:0/1190891404 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05ac0412c0 con 0x7f05b80f5410 2023-11-15T04:32:15.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.781+0000 7f05bdb44700 1 --2- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f05a402ff60 0x7f05a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:15.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.782+0000 7f05bdb44700 1 --2- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f05a402ff60 0x7f05a4032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f05b40096a0 tx=0x7f05b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:15.786 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.786+0000 7f05aaffd700 1 -- 172.21.15.190:0/1190891404 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05ac01c970 con 0x7f05b80f5410 2023-11-15T04:32:15.996 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.996+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05b80faf50 con 0x7f05b80f5410 2023-11-15T04:32:15.996 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.996+0000 7f05aaffd700 1 -- 172.21.15.190:0/1190891404 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f05ac01c970 con 0x7f05b80f5410 2023-11-15T04:32:15.997 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:15.997 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:15.998 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f05a402ff60 msgr2=0x7f05a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:15.998 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 --2- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f05a402ff60 0x7f05a4032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f05b40096a0 tx=0x7f05b4006b40).stop 2023-11-15T04:32:15.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 msgr2=0x7f05b8108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:15.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 --2- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 0x7f05b8108910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f05ac013510 tx=0x7f05ac00b040).stop 2023-11-15T04:32:15.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 shutdown_connections 2023-11-15T04:32:15.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 --2- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f05a402ff60 0x7f05a4032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:15.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 --2- 172.21.15.190:0/1190891404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05b80f5410 0x7f05b8108910 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:16.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 >> 172.21.15.190:0/1190891404 conn(0x7f05b80f0b70 msgr2=0x7f05b805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:16.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 shutdown_connections 2023-11-15T04:32:16.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:15.998+0000 7f05c05a9700 1 -- 172.21.15.190:0/1190891404 wait complete. 2023-11-15T04:32:16.000 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:15.085133+0000 mgr.smithi047.zgysuz (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:16.669 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:15.997043+0000 mon.smithi047 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.190:0/1190891404' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:17.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:17.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:18.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:17.085419+0000 mgr.smithi047.zgysuz (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:18.524 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.523+0000 7fe9a0906700 1 -- 172.21.15.190:0/2229751118 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 msgr2=0x7fe99c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:18.525 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.523+0000 7fe9a0906700 1 --2- 172.21.15.190:0/2229751118 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 0x7fe99c0f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fe990009fb0 tx=0x7fe990009b30).stop 2023-11-15T04:32:18.525 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.523+0000 7fe9a0906700 1 -- 172.21.15.190:0/2229751118 shutdown_connections 2023-11-15T04:32:18.525 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.523+0000 7fe9a0906700 1 --2- 172.21.15.190:0/2229751118 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 0x7fe99c0f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:18.525 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.523+0000 7fe9a0906700 1 -- 172.21.15.190:0/2229751118 >> 172.21.15.190:0/2229751118 conn(0x7fe99c0f0b70 msgr2=0x7fe99c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:18.525 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.523+0000 7fe9a0906700 1 -- 172.21.15.190:0/2229751118 shutdown_connections 2023-11-15T04:32:18.526 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.523+0000 7fe9a0906700 1 -- 172.21.15.190:0/2229751118 wait complete. 2023-11-15T04:32:18.526 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.524+0000 7fe9a0906700 1 Processor -- start 2023-11-15T04:32:18.526 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.524+0000 7fe9a0906700 1 -- start start 2023-11-15T04:32:18.526 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.524+0000 7fe9a0906700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 0x7fe99c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:18.527 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.524+0000 7fe9a0906700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c105b20 con 0x7fe99c0f5410 2023-11-15T04:32:18.527 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.524+0000 7fe99a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 0x7fe99c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:18.527 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.524+0000 7fe99a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 0x7fe99c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56254/0 (socket says 172.21.15.190:56254) 2023-11-15T04:32:18.527 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.524+0000 7fe99a59c700 1 -- 172.21.15.190:0/3574077626 learned_addr learned my addr 172.21.15.190:0/3574077626 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:18.528 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.525+0000 7fe99a59c700 1 -- 172.21.15.190:0/3574077626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe990005040 con 0x7fe99c0f5410 2023-11-15T04:32:18.528 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.525+0000 7fe99a59c700 1 --2- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 0x7fe99c105630 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe990013510 tx=0x7fe99000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:18.529 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.525+0000 7fe98affd700 1 -- 172.21.15.190:0/3574077626 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe990009320 con 0x7fe99c0f5410 2023-11-15T04:32:18.529 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.525+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99c105d20 con 0x7fe99c0f5410 2023-11-15T04:32:18.530 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.525+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99c1061b0 con 0x7fe99c0f5410 2023-11-15T04:32:18.530 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.525+0000 7fe98affd700 1 -- 172.21.15.190:0/3574077626 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe990013f00 con 0x7fe99c0f5410 2023-11-15T04:32:18.530 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.525+0000 7fe98affd700 1 -- 172.21.15.190:0/3574077626 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe990014ce0 con 0x7fe99c0f5410 2023-11-15T04:32:18.530 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.526+0000 7fe98affd700 1 -- 172.21.15.190:0/3574077626 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe990014420 con 0x7fe99c0f5410 2023-11-15T04:32:18.530 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.526+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe980004fa0 con 0x7fe99c0f5410 2023-11-15T04:32:18.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.526+0000 7fe98affd700 1 --2- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe98402ff60 0x7fe984032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:18.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.526+0000 7fe98affd700 1 -- 172.21.15.190:0/3574077626 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9900412d0 con 0x7fe99c0f5410 2023-11-15T04:32:18.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.526+0000 7fe999d9b700 1 --2- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe98402ff60 0x7fe984032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:18.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.527+0000 7fe999d9b700 1 --2- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe98402ff60 0x7fe984032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fe98c0096a0 tx=0x7fe98c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:18.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.531+0000 7fe98affd700 1 -- 172.21.15.190:0/3574077626 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe99000f990 con 0x7fe99c0f5410 2023-11-15T04:32:18.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.736+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe980005670 con 0x7fe99c0f5410 2023-11-15T04:32:18.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.737+0000 7fe98affd700 1 -- 172.21.15.190:0/3574077626 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe99000f990 con 0x7fe99c0f5410 2023-11-15T04:32:18.737 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:18.737 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:18.738 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe98402ff60 msgr2=0x7fe984032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:18.739 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 --2- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe98402ff60 0x7fe984032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fe98c0096a0 tx=0x7fe98c006b40).stop 2023-11-15T04:32:18.739 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 msgr2=0x7fe99c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:18.739 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 --2- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 0x7fe99c105630 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe990013510 tx=0x7fe99000b040).stop 2023-11-15T04:32:18.739 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 shutdown_connections 2023-11-15T04:32:18.739 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 --2- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe98402ff60 0x7fe984032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:18.740 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 --2- 172.21.15.190:0/3574077626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe99c0f5410 0x7fe99c105630 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:18.740 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 >> 172.21.15.190:0/3574077626 conn(0x7fe99c0f0b70 msgr2=0x7fe99c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:18.740 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 shutdown_connections 2023-11-15T04:32:18.740 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:18.739+0000 7fe9a0906700 1 -- 172.21.15.190:0/3574077626 wait complete. 2023-11-15T04:32:18.741 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:19.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:18.737407+0000 mon.smithi047 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.190:0/3574077626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:20.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:20.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:20.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:19 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:19.085729+0000 mgr.smithi047.zgysuz (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:21.204 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.202+0000 7f73d6868700 1 -- 172.21.15.190:0/1030594479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 msgr2=0x7f73d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:21.204 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.202+0000 7f73d6868700 1 --2- 172.21.15.190:0/1030594479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 0x7f73d00f6930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f73c0009fb0 tx=0x7f73c0009b30).stop 2023-11-15T04:32:21.204 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.202+0000 7f73d6868700 1 -- 172.21.15.190:0/1030594479 shutdown_connections 2023-11-15T04:32:21.204 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.202+0000 7f73d6868700 1 --2- 172.21.15.190:0/1030594479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 0x7f73d00f6930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.202+0000 7f73d6868700 1 -- 172.21.15.190:0/1030594479 >> 172.21.15.190:0/1030594479 conn(0x7f73d00f1c70 msgr2=0x7f73d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.202+0000 7f73d6868700 1 -- 172.21.15.190:0/1030594479 shutdown_connections 2023-11-15T04:32:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.202+0000 7f73d6868700 1 -- 172.21.15.190:0/1030594479 wait complete. 2023-11-15T04:32:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.203+0000 7f73d6868700 1 Processor -- start 2023-11-15T04:32:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.203+0000 7f73d6868700 1 -- start start 2023-11-15T04:32:21.206 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.203+0000 7f73d6868700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 0x7f73d0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:21.206 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.203+0000 7f73d6868700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0106c20 con 0x7f73d00f6510 2023-11-15T04:32:21.206 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.203+0000 7f73cffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 0x7f73d0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:21.206 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.203+0000 7f73cffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 0x7f73d0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34006/0 (socket says 172.21.15.190:34006) 2023-11-15T04:32:21.206 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.203+0000 7f73cffff700 1 -- 172.21.15.190:0/2245320567 learned_addr learned my addr 172.21.15.190:0/2245320567 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:21.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.204+0000 7f73cffff700 1 -- 172.21.15.190:0/2245320567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0005040 con 0x7f73d00f6510 2023-11-15T04:32:21.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.204+0000 7f73cffff700 1 --2- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 0x7f73d0106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f73c0013510 tx=0x7f73c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:21.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.204+0000 7f73ccff9700 1 -- 172.21.15.190:0/2245320567 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73c0009320 con 0x7f73d00f6510 2023-11-15T04:32:21.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.204+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d0106e20 con 0x7f73d00f6510 2023-11-15T04:32:21.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.204+0000 7f73ccff9700 1 -- 172.21.15.190:0/2245320567 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73c0013f00 con 0x7f73d00f6510 2023-11-15T04:32:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.204+0000 7f73ccff9700 1 -- 172.21.15.190:0/2245320567 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73c0014d90 con 0x7f73d00f6510 2023-11-15T04:32:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.204+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d01072b0 con 0x7f73d00f6510 2023-11-15T04:32:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.205+0000 7f73ccff9700 1 -- 172.21.15.190:0/2245320567 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73c00095f0 con 0x7f73d00f6510 2023-11-15T04:32:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.205+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d0045ad0 con 0x7f73d00f6510 2023-11-15T04:32:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.205+0000 7f73ccff9700 1 --2- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73b802ff60 0x7f73b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.205+0000 7f73ccff9700 1 -- 172.21.15.190:0/2245320567 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f73c0040fd0 con 0x7f73d00f6510 2023-11-15T04:32:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.205+0000 7f73cf7fe700 1 --2- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73b802ff60 0x7f73b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.206+0000 7f73cf7fe700 1 --2- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73b802ff60 0x7f73b8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f73c40096a0 tx=0x7f73c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.210+0000 7f73ccff9700 1 -- 172.21.15.190:0/2245320567 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73c0014420 con 0x7f73d00f6510 2023-11-15T04:32:21.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.419+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73d00fbf60 con 0x7f73d00f6510 2023-11-15T04:32:21.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.420+0000 7f73ccff9700 1 -- 172.21.15.190:0/2245320567 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f73c001d7f0 con 0x7f73d00f6510 2023-11-15T04:32:21.420 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:21.421 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:21.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73b802ff60 msgr2=0x7f73b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:21.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 --2- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73b802ff60 0x7f73b8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f73c40096a0 tx=0x7f73c4006b40).stop 2023-11-15T04:32:21.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 msgr2=0x7f73d0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:21.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 --2- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 0x7f73d0106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f73c0013510 tx=0x7f73c000b040).stop 2023-11-15T04:32:21.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 shutdown_connections 2023-11-15T04:32:21.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 --2- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73b802ff60 0x7f73b8032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:21.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 --2- 172.21.15.190:0/2245320567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d00f6510 0x7f73d0106730 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:21.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 >> 172.21.15.190:0/2245320567 conn(0x7f73d00f1c70 msgr2=0x7f73d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:21.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 shutdown_connections 2023-11-15T04:32:21.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:21.422+0000 7f73d6868700 1 -- 172.21.15.190:0/2245320567 wait complete. 2023-11-15T04:32:21.424 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:21.086101+0000 mgr.smithi047.zgysuz (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:21.420649+0000 mon.smithi047 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.190:0/2245320567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:22.840 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:22.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:23.998 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.998+0000 7fbe8e053700 1 -- 172.21.15.190:0/1960797156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 msgr2=0x7fbe880f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:23.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.998+0000 7fbe8e053700 1 --2- 172.21.15.190:0/1960797156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 0x7fbe880f9170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fbe70009fb0 tx=0x7fbe70009b30).stop 2023-11-15T04:32:23.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.998+0000 7fbe8e053700 1 -- 172.21.15.190:0/1960797156 shutdown_connections 2023-11-15T04:32:23.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.998+0000 7fbe8e053700 1 --2- 172.21.15.190:0/1960797156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 0x7fbe880f9170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:24.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.998+0000 7fbe8e053700 1 -- 172.21.15.190:0/1960797156 >> 172.21.15.190:0/1960797156 conn(0x7fbe880f0b70 msgr2=0x7fbe880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:24.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.998+0000 7fbe8e053700 1 -- 172.21.15.190:0/1960797156 shutdown_connections 2023-11-15T04:32:24.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.998+0000 7fbe8e053700 1 -- 172.21.15.190:0/1960797156 wait complete. 2023-11-15T04:32:24.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.999+0000 7fbe8e053700 1 Processor -- start 2023-11-15T04:32:24.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.999+0000 7fbe8e053700 1 -- start start 2023-11-15T04:32:24.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.999+0000 7fbe8e053700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 0x7fbe881066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:24.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.999+0000 7fbe8e053700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe88106b90 con 0x7fbe88060200 2023-11-15T04:32:24.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.999+0000 7fbe877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 0x7fbe881066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:24.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.999+0000 7fbe877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 0x7fbe881066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34030/0 (socket says 172.21.15.190:34030) 2023-11-15T04:32:24.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.999+0000 7fbe877fe700 1 -- 172.21.15.190:0/611716942 learned_addr learned my addr 172.21.15.190:0/611716942 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:24.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:23.999+0000 7fbe877fe700 1 -- 172.21.15.190:0/611716942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70005040 con 0x7fbe88060200 2023-11-15T04:32:24.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.000+0000 7fbe877fe700 1 --2- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 0x7fbe881066a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbe70013510 tx=0x7fbe7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:24.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.000+0000 7fbe7ffff700 1 -- 172.21.15.190:0/611716942 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe70009320 con 0x7fbe88060200 2023-11-15T04:32:24.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.000+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe88106d90 con 0x7fbe88060200 2023-11-15T04:32:24.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.000+0000 7fbe7ffff700 1 -- 172.21.15.190:0/611716942 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe70013f00 con 0x7fbe88060200 2023-11-15T04:32:24.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.000+0000 7fbe7ffff700 1 -- 172.21.15.190:0/611716942 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe70014dc0 con 0x7fbe88060200 2023-11-15T04:32:24.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.000+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe88107220 con 0x7fbe88060200 2023-11-15T04:32:24.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.001+0000 7fbe7ffff700 1 -- 172.21.15.190:0/611716942 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe70014420 con 0x7fbe88060200 2023-11-15T04:32:24.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.001+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe88045ad0 con 0x7fbe88060200 2023-11-15T04:32:24.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.001+0000 7fbe7ffff700 1 --2- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe7402ff60 0x7fbe74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:24.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.001+0000 7fbe7ffff700 1 -- 172.21.15.190:0/611716942 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe70041110 con 0x7fbe88060200 2023-11-15T04:32:24.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.001+0000 7fbe86ffd700 1 --2- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe7402ff60 0x7fbe74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:24.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.002+0000 7fbe86ffd700 1 --2- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe7402ff60 0x7fbe74032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbe780096a0 tx=0x7fbe78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:24.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.006+0000 7fbe7ffff700 1 -- 172.21.15.190:0/611716942 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe700095f0 con 0x7fbe88060200 2023-11-15T04:32:24.210 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.210+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe881080d0 con 0x7fbe88060200 2023-11-15T04:32:24.210 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.210+0000 7fbe7ffff700 1 -- 172.21.15.190:0/611716942 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe7001cb10 con 0x7fbe88060200 2023-11-15T04:32:24.211 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:24.211 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:24.212 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe7402ff60 msgr2=0x7fbe74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:24.212 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 --2- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe7402ff60 0x7fbe74032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbe780096a0 tx=0x7fbe78006b40).stop 2023-11-15T04:32:24.212 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 msgr2=0x7fbe881066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:24.212 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 --2- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 0x7fbe881066a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbe70013510 tx=0x7fbe7000b040).stop 2023-11-15T04:32:24.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 shutdown_connections 2023-11-15T04:32:24.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 --2- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbe7402ff60 0x7fbe74032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:24.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 --2- 172.21.15.190:0/611716942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe88060200 0x7fbe881066a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:24.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 >> 172.21.15.190:0/611716942 conn(0x7fbe880f0b70 msgr2=0x7fbe880f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:24.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 shutdown_connections 2023-11-15T04:32:24.214 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:24.212+0000 7fbe8e053700 1 -- 172.21.15.190:0/611716942 wait complete. 2023-11-15T04:32:24.214 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:24.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:23.086474+0000 mgr.smithi047.zgysuz (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:25.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:25 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:24.211083+0000 mon.smithi047 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.190:0/611716942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:25.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:25.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:26.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:25.086742+0000 mgr.smithi047.zgysuz (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:26.792 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.790+0000 7febfc8df700 1 -- 172.21.15.190:0/1124662109 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 msgr2=0x7febf80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:26.792 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.790+0000 7febfc8df700 1 --2- 172.21.15.190:0/1124662109 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 0x7febf80f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7febe8009fb0 tx=0x7febe8009b30).stop 2023-11-15T04:32:26.793 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.790+0000 7febfc8df700 1 -- 172.21.15.190:0/1124662109 shutdown_connections 2023-11-15T04:32:26.793 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.790+0000 7febfc8df700 1 --2- 172.21.15.190:0/1124662109 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 0x7febf80f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:26.793 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.790+0000 7febfc8df700 1 -- 172.21.15.190:0/1124662109 >> 172.21.15.190:0/1124662109 conn(0x7febf80f0b70 msgr2=0x7febf80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:26.793 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.790+0000 7febfc8df700 1 -- 172.21.15.190:0/1124662109 shutdown_connections 2023-11-15T04:32:26.793 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.790+0000 7febfc8df700 1 -- 172.21.15.190:0/1124662109 wait complete. 2023-11-15T04:32:26.794 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.791+0000 7febfc8df700 1 Processor -- start 2023-11-15T04:32:26.794 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.791+0000 7febfc8df700 1 -- start start 2023-11-15T04:32:26.794 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.791+0000 7febfc8df700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 0x7febf8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:26.794 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.791+0000 7febfc8df700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf8060220 con 0x7febf80f5410 2023-11-15T04:32:26.795 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.791+0000 7febf659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 0x7febf8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:26.795 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.791+0000 7febf659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 0x7febf8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34054/0 (socket says 172.21.15.190:34054) 2023-11-15T04:32:26.795 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.791+0000 7febf659c700 1 -- 172.21.15.190:0/3597974337 learned_addr learned my addr 172.21.15.190:0/3597974337 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:26.798 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.792+0000 7febf659c700 1 -- 172.21.15.190:0/3597974337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe8005040 con 0x7febf80f5410 2023-11-15T04:32:26.798 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.792+0000 7febf659c700 1 --2- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 0x7febf8109450 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7febe8013510 tx=0x7febe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:26.798 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.792+0000 7febeeffd700 1 -- 172.21.15.190:0/3597974337 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febe8009320 con 0x7febf80f5410 2023-11-15T04:32:26.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.792+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf8060420 con 0x7febf80f5410 2023-11-15T04:32:26.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.792+0000 7febeeffd700 1 -- 172.21.15.190:0/3597974337 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febe8013f00 con 0x7febf80f5410 2023-11-15T04:32:26.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.792+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf8109b00 con 0x7febf80f5410 2023-11-15T04:32:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.792+0000 7febeeffd700 1 -- 172.21.15.190:0/3597974337 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febe8014d90 con 0x7febf80f5410 2023-11-15T04:32:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.793+0000 7febeeffd700 1 -- 172.21.15.190:0/3597974337 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7febe80263a0 con 0x7febf80f5410 2023-11-15T04:32:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.793+0000 7febeeffd700 1 --2- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7febe402ffb0 0x7febe4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.793+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febdc004fa0 con 0x7febf80f5410 2023-11-15T04:32:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.793+0000 7febeeffd700 1 -- 172.21.15.190:0/3597974337 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7febe8040f10 con 0x7febf80f5410 2023-11-15T04:32:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.793+0000 7febf5d9b700 1 --2- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7febe402ffb0 0x7febe4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.794+0000 7febf5d9b700 1 --2- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7febe402ffb0 0x7febe4032470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7febe00096a0 tx=0x7febe0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:26.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:26.798+0000 7febeeffd700 1 -- 172.21.15.190:0/3597974337 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febe8012920 con 0x7febf80f5410 2023-11-15T04:32:27.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.003+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febdc005670 con 0x7febf80f5410 2023-11-15T04:32:27.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.003+0000 7febeeffd700 1 -- 172.21.15.190:0/3597974337 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7febe802b020 con 0x7febf80f5410 2023-11-15T04:32:27.005 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:27.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7febe402ffb0 msgr2=0x7febe4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:27.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 --2- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7febe402ffb0 0x7febe4032470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7febe00096a0 tx=0x7febe0006b40).stop 2023-11-15T04:32:27.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 msgr2=0x7febf8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:27.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 --2- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 0x7febf8109450 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7febe8013510 tx=0x7febe800b040).stop 2023-11-15T04:32:27.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 shutdown_connections 2023-11-15T04:32:27.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 --2- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7febe402ffb0 0x7febe4032470 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:27.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 --2- 172.21.15.190:0/3597974337 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80f5410 0x7febf8109450 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:27.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 >> 172.21.15.190:0/3597974337 conn(0x7febf80f0b70 msgr2=0x7febf80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:27.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 shutdown_connections 2023-11-15T04:32:27.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:27.005+0000 7febfc8df700 1 -- 172.21.15.190:0/3597974337 wait complete. 2023-11-15T04:32:27.008 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:27.004809+0000 mon.smithi047 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.190:0/3597974337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:27.087113+0000 mgr.smithi047.zgysuz (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:28.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:28.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:29.522 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.517+0000 7f58870f1700 1 -- 172.21.15.190:0/209690512 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 msgr2=0x7f58800f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:29.522 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.517+0000 7f58870f1700 1 --2- 172.21.15.190:0/209690512 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 0x7f58800f8f20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f587c009fb0 tx=0x7f587c009b30).stop 2023-11-15T04:32:29.522 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.517+0000 7f58870f1700 1 -- 172.21.15.190:0/209690512 shutdown_connections 2023-11-15T04:32:29.523 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.517+0000 7f58870f1700 1 --2- 172.21.15.190:0/209690512 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 0x7f58800f8f20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:29.523 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.517+0000 7f58870f1700 1 -- 172.21.15.190:0/209690512 >> 172.21.15.190:0/209690512 conn(0x7f58800f09a0 msgr2=0x7f58800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:29.524 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.517+0000 7f58870f1700 1 -- 172.21.15.190:0/209690512 shutdown_connections 2023-11-15T04:32:29.524 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.517+0000 7f58870f1700 1 -- 172.21.15.190:0/209690512 wait complete. 2023-11-15T04:32:29.524 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.517+0000 7f58870f1700 1 Processor -- start 2023-11-15T04:32:29.525 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.518+0000 7f58870f1700 1 -- start start 2023-11-15T04:32:29.525 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.518+0000 7f58870f1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 0x7f5880106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:29.526 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.518+0000 7f58870f1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58801069f0 con 0x7f58800601e0 2023-11-15T04:32:29.526 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.518+0000 7f58860ef700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 0x7f5880106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:29.526 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.518+0000 7f58860ef700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 0x7f5880106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34072/0 (socket says 172.21.15.190:34072) 2023-11-15T04:32:29.527 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.518+0000 7f58860ef700 1 -- 172.21.15.190:0/2845304727 learned_addr learned my addr 172.21.15.190:0/2845304727 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:29.527 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.518+0000 7f58860ef700 1 -- 172.21.15.190:0/2845304727 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c005040 con 0x7f58800601e0 2023-11-15T04:32:29.527 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.519+0000 7f58860ef700 1 --2- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 0x7f5880106500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f587c013510 tx=0x7f587c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:29.528 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.519+0000 7f5876ffd700 1 -- 172.21.15.190:0/2845304727 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f587c009320 con 0x7f58800601e0 2023-11-15T04:32:29.528 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.519+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5880106bf0 con 0x7f58800601e0 2023-11-15T04:32:29.529 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.519+0000 7f5876ffd700 1 -- 172.21.15.190:0/2845304727 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f587c013f00 con 0x7f58800601e0 2023-11-15T04:32:29.529 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.519+0000 7f5876ffd700 1 -- 172.21.15.190:0/2845304727 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f587c014d90 con 0x7f58800601e0 2023-11-15T04:32:29.530 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.519+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5880107080 con 0x7f58800601e0 2023-11-15T04:32:29.530 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.520+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5880045ad0 con 0x7f58800601e0 2023-11-15T04:32:29.530 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.520+0000 7f5876ffd700 1 -- 172.21.15.190:0/2845304727 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f587c0095f0 con 0x7f58800601e0 2023-11-15T04:32:29.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.520+0000 7f5876ffd700 1 --2- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f586c02ff60 0x7f586c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:29.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.520+0000 7f5876ffd700 1 -- 172.21.15.190:0/2845304727 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f587c040fc0 con 0x7f58800601e0 2023-11-15T04:32:29.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.520+0000 7f58858ee700 1 --2- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f586c02ff60 0x7f586c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:29.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.521+0000 7f58858ee700 1 --2- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f586c02ff60 0x7f586c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f58700096a0 tx=0x7f5870006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:29.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.524+0000 7f5876ffd700 1 -- 172.21.15.190:0/2845304727 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f587c014420 con 0x7f58800601e0 2023-11-15T04:32:29.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.730+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5880107ed0 con 0x7f58800601e0 2023-11-15T04:32:29.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.731+0000 7f5876ffd700 1 -- 172.21.15.190:0/2845304727 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f587c010230 con 0x7f58800601e0 2023-11-15T04:32:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:29.732 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:29.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.732+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f586c02ff60 msgr2=0x7f586c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:29.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.732+0000 7f58870f1700 1 --2- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f586c02ff60 0x7f586c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f58700096a0 tx=0x7f5870006b40).stop 2023-11-15T04:32:29.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.732+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 msgr2=0x7f5880106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:29.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.732+0000 7f58870f1700 1 --2- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 0x7f5880106500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f587c013510 tx=0x7f587c00b040).stop 2023-11-15T04:32:29.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.732+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 shutdown_connections 2023-11-15T04:32:29.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.732+0000 7f58870f1700 1 --2- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f586c02ff60 0x7f586c032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:29.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.732+0000 7f58870f1700 1 --2- 172.21.15.190:0/2845304727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58800601e0 0x7f5880106500 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:29.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.732+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 >> 172.21.15.190:0/2845304727 conn(0x7f58800f09a0 msgr2=0x7f58800f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:29.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.733+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 shutdown_connections 2023-11-15T04:32:29.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:29.733+0000 7f58870f1700 1 -- 172.21.15.190:0/2845304727 wait complete. 2023-11-15T04:32:29.737 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:30.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:29.087443+0000 mgr.smithi047.zgysuz (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:30.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:29.732154+0000 mon.smithi047 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.190:0/2845304727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:31.115 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:31.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:32.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.253+0000 7f9a32c4b700 1 -- 172.21.15.190:0/1980129649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 msgr2=0x7f9a2c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.253+0000 7f9a32c4b700 1 --2- 172.21.15.190:0/1980129649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 0x7f9a2c0f55c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f9a28009fb0 tx=0x7f9a28009b30).stop 2023-11-15T04:32:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.253+0000 7f9a32c4b700 1 -- 172.21.15.190:0/1980129649 shutdown_connections 2023-11-15T04:32:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.253+0000 7f9a32c4b700 1 --2- 172.21.15.190:0/1980129649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 0x7f9a2c0f55c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.253+0000 7f9a32c4b700 1 -- 172.21.15.190:0/1980129649 >> 172.21.15.190:0/1980129649 conn(0x7f9a2c0f09a0 msgr2=0x7f9a2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.253+0000 7f9a32c4b700 1 -- 172.21.15.190:0/1980129649 shutdown_connections 2023-11-15T04:32:32.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.253+0000 7f9a32c4b700 1 -- 172.21.15.190:0/1980129649 wait complete. 2023-11-15T04:32:32.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.254+0000 7f9a32c4b700 1 Processor -- start 2023-11-15T04:32:32.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.254+0000 7f9a32c4b700 1 -- start start 2023-11-15T04:32:32.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.254+0000 7f9a32c4b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 0x7f9a2c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:32.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.254+0000 7f9a32c4b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c0fb5a0 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.254+0000 7f9a31c49700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 0x7f9a2c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.254+0000 7f9a31c49700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 0x7f9a2c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34452/0 (socket says 172.21.15.190:34452) 2023-11-15T04:32:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.254+0000 7f9a31c49700 1 -- 172.21.15.190:0/3479213752 learned_addr learned my addr 172.21.15.190:0/3479213752 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.254+0000 7f9a31c49700 1 -- 172.21.15.190:0/3479213752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a28005040 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.254+0000 7f9a31c49700 1 --2- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 0x7f9a2c0fd120 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f9a28013510 tx=0x7f9a2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:32.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.255+0000 7f9a227fc700 1 -- 172.21.15.190:0/3479213752 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a28009320 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.255+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c0fb740 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.255+0000 7f9a227fc700 1 -- 172.21.15.190:0/3479213752 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a28013f00 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.255+0000 7f9a227fc700 1 -- 172.21.15.190:0/3479213752 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a28014dc0 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.255+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c0fbbd0 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.255+0000 7f9a227fc700 1 -- 172.21.15.190:0/3479213752 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a280095f0 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.255+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2c059100 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.256+0000 7f9a227fc700 1 --2- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9a1802ff60 0x7f9a18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:32.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.256+0000 7f9a227fc700 1 -- 172.21.15.190:0/3479213752 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a28041000 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.256+0000 7f9a31448700 1 --2- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9a1802ff60 0x7f9a18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:32.262 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.256+0000 7f9a31448700 1 --2- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9a1802ff60 0x7f9a18032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f9a1c0096a0 tx=0x7f9a1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:32.262 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.260+0000 7f9a227fc700 1 -- 172.21.15.190:0/3479213752 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a28014420 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.469 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.469+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a2c0fabe0 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.470 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.469+0000 7f9a227fc700 1 -- 172.21.15.190:0/3479213752 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a28010230 con 0x7f9a2c0f51a0 2023-11-15T04:32:32.471 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:32.471 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:32.472 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9a1802ff60 msgr2=0x7f9a18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:32.472 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 --2- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9a1802ff60 0x7f9a18032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f9a1c0096a0 tx=0x7f9a1c006b40).stop 2023-11-15T04:32:32.472 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 msgr2=0x7f9a2c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:32.473 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 --2- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 0x7f9a2c0fd120 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f9a28013510 tx=0x7f9a2800b040).stop 2023-11-15T04:32:32.473 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 shutdown_connections 2023-11-15T04:32:32.473 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 --2- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f9a1802ff60 0x7f9a18032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:32.473 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 --2- 172.21.15.190:0/3479213752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a2c0f51a0 0x7f9a2c0fd120 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:32.473 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 >> 172.21.15.190:0/3479213752 conn(0x7f9a2c0f09a0 msgr2=0x7f9a2c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:32.474 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 shutdown_connections 2023-11-15T04:32:32.474 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:32.471+0000 7f9a32c4b700 1 -- 172.21.15.190:0/3479213752 wait complete. 2023-11-15T04:32:32.474 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:31.087820+0000 mgr.smithi047.zgysuz (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:33.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:33 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:32.470888+0000 mon.smithi047 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.190:0/3479213752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:33.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:33.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:34.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:33.088199+0000 mgr.smithi047.zgysuz (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:35.064 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.061+0000 7f407512f700 1 -- 172.21.15.190:0/1624965751 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 msgr2=0x7f40700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:35.064 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.061+0000 7f407512f700 1 --2- 172.21.15.190:0/1624965751 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 0x7f40700f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f4064009fb0 tx=0x7f4064009b30).stop 2023-11-15T04:32:35.064 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.061+0000 7f407512f700 1 -- 172.21.15.190:0/1624965751 shutdown_connections 2023-11-15T04:32:35.064 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.061+0000 7f407512f700 1 --2- 172.21.15.190:0/1624965751 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 0x7f40700f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:35.065 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.061+0000 7f407512f700 1 -- 172.21.15.190:0/1624965751 >> 172.21.15.190:0/1624965751 conn(0x7f40700f0b70 msgr2=0x7f40700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:35.065 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.061+0000 7f407512f700 1 -- 172.21.15.190:0/1624965751 shutdown_connections 2023-11-15T04:32:35.065 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.061+0000 7f407512f700 1 -- 172.21.15.190:0/1624965751 wait complete. 2023-11-15T04:32:35.065 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.062+0000 7f407512f700 1 Processor -- start 2023-11-15T04:32:35.065 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.062+0000 7f407512f700 1 -- start start 2023-11-15T04:32:35.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.062+0000 7f407512f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 0x7f4070106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:35.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.062+0000 7f407512f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4070106c20 con 0x7f40700f5410 2023-11-15T04:32:35.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.062+0000 7f406ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 0x7f4070106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:35.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.062+0000 7f406ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 0x7f4070106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34458/0 (socket says 172.21.15.190:34458) 2023-11-15T04:32:35.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.062+0000 7f406ed9d700 1 -- 172.21.15.190:0/3116770030 learned_addr learned my addr 172.21.15.190:0/3116770030 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:35.067 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.062+0000 7f406ed9d700 1 -- 172.21.15.190:0/3116770030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064005040 con 0x7f40700f5410 2023-11-15T04:32:35.067 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.063+0000 7f406ed9d700 1 --2- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 0x7f4070106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f4064013510 tx=0x7f406400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:35.067 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.063+0000 7f40577fe700 1 -- 172.21.15.190:0/3116770030 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4064009320 con 0x7f40700f5410 2023-11-15T04:32:35.067 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.063+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4070106e20 con 0x7f40700f5410 2023-11-15T04:32:35.067 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.063+0000 7f40577fe700 1 -- 172.21.15.190:0/3116770030 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4064013f00 con 0x7f40700f5410 2023-11-15T04:32:35.068 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.063+0000 7f40577fe700 1 -- 172.21.15.190:0/3116770030 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4064014d90 con 0x7f40700f5410 2023-11-15T04:32:35.068 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.063+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40701072b0 con 0x7f40700f5410 2023-11-15T04:32:35.069 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.064+0000 7f40577fe700 1 -- 172.21.15.190:0/3116770030 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4064014420 con 0x7f40700f5410 2023-11-15T04:32:35.070 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.064+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4070045ad0 con 0x7f40700f5410 2023-11-15T04:32:35.070 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.064+0000 7f40577fe700 1 --2- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f405802ff60 0x7f4058032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:35.070 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.064+0000 7f40577fe700 1 -- 172.21.15.190:0/3116770030 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4064040fb0 con 0x7f40700f5410 2023-11-15T04:32:35.070 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.064+0000 7f406e59c700 1 --2- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f405802ff60 0x7f4058032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:35.070 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.064+0000 7f406e59c700 1 --2- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f405802ff60 0x7f4058032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f40600096a0 tx=0x7f4060006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:35.071 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.069+0000 7f40577fe700 1 -- 172.21.15.190:0/3116770030 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40640095f0 con 0x7f40700f5410 2023-11-15T04:32:35.277 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.276+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40700fae60 con 0x7f40700f5410 2023-11-15T04:32:35.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.277+0000 7f40577fe700 1 -- 172.21.15.190:0/3116770030 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f406401db70 con 0x7f40700f5410 2023-11-15T04:32:35.278 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:35.278 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:35.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.278+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f405802ff60 msgr2=0x7f4058032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:35.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.278+0000 7f407512f700 1 --2- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f405802ff60 0x7f4058032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f40600096a0 tx=0x7f4060006b40).stop 2023-11-15T04:32:35.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.278+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 msgr2=0x7f4070106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:35.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.278+0000 7f407512f700 1 --2- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 0x7f4070106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f4064013510 tx=0x7f406400b040).stop 2023-11-15T04:32:35.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.278+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 shutdown_connections 2023-11-15T04:32:35.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.279+0000 7f407512f700 1 --2- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f405802ff60 0x7f4058032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:35.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.279+0000 7f407512f700 1 --2- 172.21.15.190:0/3116770030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40700f5410 0x7f4070106730 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:35.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.279+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 >> 172.21.15.190:0/3116770030 conn(0x7f40700f0b70 msgr2=0x7f40700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:35.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.279+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 shutdown_connections 2023-11-15T04:32:35.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:35.279+0000 7f407512f700 1 -- 172.21.15.190:0/3116770030 wait complete. 2023-11-15T04:32:35.281 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:35.088581+0000 mgr.smithi047.zgysuz (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:35.278520+0000 mon.smithi047 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.190:0/3116770030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:36.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:36.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:37.809 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.807+0000 7efe456b6700 1 -- 172.21.15.190:0/3144319018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 msgr2=0x7efe400f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:37.809 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.807+0000 7efe456b6700 1 --2- 172.21.15.190:0/3144319018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 0x7efe400f4be0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7efe34009fb0 tx=0x7efe34009b30).stop 2023-11-15T04:32:37.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.807+0000 7efe456b6700 1 -- 172.21.15.190:0/3144319018 shutdown_connections 2023-11-15T04:32:37.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.807+0000 7efe456b6700 1 --2- 172.21.15.190:0/3144319018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 0x7efe400f4be0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:37.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.807+0000 7efe456b6700 1 -- 172.21.15.190:0/3144319018 >> 172.21.15.190:0/3144319018 conn(0x7efe400f02e0 msgr2=0x7efe400f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:37.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.807+0000 7efe456b6700 1 -- 172.21.15.190:0/3144319018 shutdown_connections 2023-11-15T04:32:37.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.807+0000 7efe456b6700 1 -- 172.21.15.190:0/3144319018 wait complete. 2023-11-15T04:32:37.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.808+0000 7efe456b6700 1 Processor -- start 2023-11-15T04:32:37.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.808+0000 7efe456b6700 1 -- start start 2023-11-15T04:32:37.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.808+0000 7efe456b6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 0x7efe40106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:37.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.808+0000 7efe456b6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe40106bf0 con 0x7efe400f47c0 2023-11-15T04:32:37.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe3effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 0x7efe40106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:37.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe3effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 0x7efe40106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34478/0 (socket says 172.21.15.190:34478) 2023-11-15T04:32:37.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe3effd700 1 -- 172.21.15.190:0/265874075 learned_addr learned my addr 172.21.15.190:0/265874075 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:37.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe3effd700 1 -- 172.21.15.190:0/265874075 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe34005040 con 0x7efe400f47c0 2023-11-15T04:32:37.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe3effd700 1 --2- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 0x7efe40106700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7efe34013510 tx=0x7efe3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:37.816 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe277fe700 1 -- 172.21.15.190:0/265874075 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe34009320 con 0x7efe400f47c0 2023-11-15T04:32:37.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe40106df0 con 0x7efe400f47c0 2023-11-15T04:32:37.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe277fe700 1 -- 172.21.15.190:0/265874075 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe34013f00 con 0x7efe400f47c0 2023-11-15T04:32:37.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe277fe700 1 -- 172.21.15.190:0/265874075 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe34014d90 con 0x7efe400f47c0 2023-11-15T04:32:37.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.809+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe40107280 con 0x7efe400f47c0 2023-11-15T04:32:37.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.810+0000 7efe277fe700 1 -- 172.21.15.190:0/265874075 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe34014420 con 0x7efe400f47c0 2023-11-15T04:32:37.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.810+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe40045ad0 con 0x7efe400f47c0 2023-11-15T04:32:37.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.810+0000 7efe277fe700 1 --2- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2802ff60 0x7efe28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:37.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.810+0000 7efe277fe700 1 -- 172.21.15.190:0/265874075 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe34041070 con 0x7efe400f47c0 2023-11-15T04:32:37.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.811+0000 7efe3e7fc700 1 --2- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2802ff60 0x7efe28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:37.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.811+0000 7efe3e7fc700 1 --2- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2802ff60 0x7efe28032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7efe300096a0 tx=0x7efe30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:37.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:37.815+0000 7efe277fe700 1 -- 172.21.15.190:0/265874075 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe340095f0 con 0x7efe400f47c0 2023-11-15T04:32:38.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:37.088926+0000 mgr.smithi047.zgysuz (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:38.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.024+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe40108130 con 0x7efe400f47c0 2023-11-15T04:32:38.026 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.025+0000 7efe277fe700 1 -- 172.21.15.190:0/265874075 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe3401db70 con 0x7efe400f47c0 2023-11-15T04:32:38.026 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:38.026 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:38.027 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2802ff60 msgr2=0x7efe28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:38.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 --2- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2802ff60 0x7efe28032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7efe300096a0 tx=0x7efe30006b40).stop 2023-11-15T04:32:38.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 msgr2=0x7efe40106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:38.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 --2- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 0x7efe40106700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7efe34013510 tx=0x7efe3400b040).stop 2023-11-15T04:32:38.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 shutdown_connections 2023-11-15T04:32:38.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 --2- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2802ff60 0x7efe28032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:38.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 --2- 172.21.15.190:0/265874075 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f47c0 0x7efe40106700 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:38.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 >> 172.21.15.190:0/265874075 conn(0x7efe400f02e0 msgr2=0x7efe400f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:38.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 shutdown_connections 2023-11-15T04:32:38.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:38.027+0000 7efe456b6700 1 -- 172.21.15.190:0/265874075 wait complete. 2023-11-15T04:32:38.030 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:39.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:38 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:38.026398+0000 mon.smithi047 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.190:0/265874075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:39.405 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:39.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:40.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:39 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:39.089196+0000 mgr.smithi047.zgysuz (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:40.515 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.513+0000 7ff7157c1700 1 -- 172.21.15.190:0/2409937847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 msgr2=0x7ff7100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:40.516 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.513+0000 7ff7157c1700 1 --2- 172.21.15.190:0/2409937847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 0x7ff7100f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7ff704009fb0 tx=0x7ff704009b30).stop 2023-11-15T04:32:40.516 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.513+0000 7ff7157c1700 1 -- 172.21.15.190:0/2409937847 shutdown_connections 2023-11-15T04:32:40.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.513+0000 7ff7157c1700 1 --2- 172.21.15.190:0/2409937847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 0x7ff7100f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:40.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.513+0000 7ff7157c1700 1 -- 172.21.15.190:0/2409937847 >> 172.21.15.190:0/2409937847 conn(0x7ff7100f0b70 msgr2=0x7ff7100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:40.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.513+0000 7ff7157c1700 1 -- 172.21.15.190:0/2409937847 shutdown_connections 2023-11-15T04:32:40.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.513+0000 7ff7157c1700 1 -- 172.21.15.190:0/2409937847 wait complete. 2023-11-15T04:32:40.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.514+0000 7ff7157c1700 1 Processor -- start 2023-11-15T04:32:40.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.514+0000 7ff7157c1700 1 -- start start 2023-11-15T04:32:40.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.514+0000 7ff7157c1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 0x7ff7101093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:40.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.514+0000 7ff7157c1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff710060280 con 0x7ff7100f5410 2023-11-15T04:32:40.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.514+0000 7ff70effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 0x7ff7101093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:40.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.514+0000 7ff70effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 0x7ff7101093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53426/0 (socket says 172.21.15.190:53426) 2023-11-15T04:32:40.519 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.514+0000 7ff70effd700 1 -- 172.21.15.190:0/4047185306 learned_addr learned my addr 172.21.15.190:0/4047185306 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:40.519 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.514+0000 7ff70effd700 1 -- 172.21.15.190:0/4047185306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff704005040 con 0x7ff7100f5410 2023-11-15T04:32:40.519 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.514+0000 7ff70effd700 1 --2- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 0x7ff7101093f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7ff704013510 tx=0x7ff70400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:40.519 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.515+0000 7ff6f77fe700 1 -- 172.21.15.190:0/4047185306 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff704009320 con 0x7ff7100f5410 2023-11-15T04:32:40.519 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.515+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff710060480 con 0x7ff7100f5410 2023-11-15T04:32:40.520 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.515+0000 7ff6f77fe700 1 -- 172.21.15.190:0/4047185306 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff704013f00 con 0x7ff7100f5410 2023-11-15T04:32:40.520 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.515+0000 7ff6f77fe700 1 -- 172.21.15.190:0/4047185306 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff704014d90 con 0x7ff7100f5410 2023-11-15T04:32:40.520 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.515+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff710109b20 con 0x7ff7100f5410 2023-11-15T04:32:40.521 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.515+0000 7ff6f77fe700 1 -- 172.21.15.190:0/4047185306 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7040095f0 con 0x7ff7100f5410 2023-11-15T04:32:40.521 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.515+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff710045ad0 con 0x7ff7100f5410 2023-11-15T04:32:40.522 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.516+0000 7ff6f77fe700 1 --2- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff6f802ff60 0x7ff6f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:40.522 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.516+0000 7ff6f77fe700 1 -- 172.21.15.190:0/4047185306 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff704040fe0 con 0x7ff7100f5410 2023-11-15T04:32:40.522 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.516+0000 7ff70e7fc700 1 --2- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff6f802ff60 0x7ff6f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:40.522 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.516+0000 7ff70e7fc700 1 --2- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff6f802ff60 0x7ff6f8032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ff7000096a0 tx=0x7ff700006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:40.522 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.521+0000 7ff6f77fe700 1 -- 172.21.15.190:0/4047185306 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff70401d7f0 con 0x7ff7100f5410 2023-11-15T04:32:40.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.727+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff71010a340 con 0x7ff7100f5410 2023-11-15T04:32:40.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.727+0000 7ff6f77fe700 1 -- 172.21.15.190:0/4047185306 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff704010230 con 0x7ff7100f5410 2023-11-15T04:32:40.729 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:40.729 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:40.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff6f802ff60 msgr2=0x7ff6f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:40.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 --2- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff6f802ff60 0x7ff6f8032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ff7000096a0 tx=0x7ff700006b40).stop 2023-11-15T04:32:40.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 msgr2=0x7ff7101093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:40.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 --2- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 0x7ff7101093f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7ff704013510 tx=0x7ff70400b040).stop 2023-11-15T04:32:40.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 shutdown_connections 2023-11-15T04:32:40.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 --2- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff6f802ff60 0x7ff6f8032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:40.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 --2- 172.21.15.190:0/4047185306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7100f5410 0x7ff7101093f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:40.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 >> 172.21.15.190:0/4047185306 conn(0x7ff7100f0b70 msgr2=0x7ff7100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:40.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 shutdown_connections 2023-11-15T04:32:40.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:40.729+0000 7ff7157c1700 1 -- 172.21.15.190:0/4047185306 wait complete. 2023-11-15T04:32:40.732 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:41.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:40.728987+0000 mon.smithi047 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.190:0/4047185306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:42.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:42.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:42.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:41 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:41.089540+0000 mgr.smithi047.zgysuz (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:43.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.255+0000 7f8fb37e0700 1 -- 172.21.15.190:0/1499204351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac0f6510 msgr2=0x7f8fac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:43.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.255+0000 7f8fb37e0700 1 --2- 172.21.15.190:0/1499204351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac0f6510 0x7f8fac0f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f8f9c009fb0 tx=0x7f8f9c009b30).stop 2023-11-15T04:32:43.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.256+0000 7f8fb37e0700 1 -- 172.21.15.190:0/1499204351 shutdown_connections 2023-11-15T04:32:43.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.256+0000 7f8fb37e0700 1 --2- 172.21.15.190:0/1499204351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac0f6510 0x7f8fac0f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:43.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.256+0000 7f8fb37e0700 1 -- 172.21.15.190:0/1499204351 >> 172.21.15.190:0/1499204351 conn(0x7f8fac0f1c70 msgr2=0x7f8fac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:43.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.256+0000 7f8fb37e0700 1 -- 172.21.15.190:0/1499204351 shutdown_connections 2023-11-15T04:32:43.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.256+0000 7f8fb37e0700 1 -- 172.21.15.190:0/1499204351 wait complete. 2023-11-15T04:32:43.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.256+0000 7f8fb37e0700 1 Processor -- start 2023-11-15T04:32:43.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.256+0000 7f8fb37e0700 1 -- start start 2023-11-15T04:32:43.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.256+0000 7f8fb37e0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac06b960 0x7f8fac069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:43.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.256+0000 7f8fb37e0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac06bd80 con 0x7f8fac06b960 2023-11-15T04:32:43.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fb157c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac06b960 0x7f8fac069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:43.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fb157c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac06b960 0x7f8fac069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53450/0 (socket says 172.21.15.190:53450) 2023-11-15T04:32:43.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fb157c700 1 -- 172.21.15.190:0/2972790455 learned_addr learned my addr 172.21.15.190:0/2972790455 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:43.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fb157c700 1 -- 172.21.15.190:0/2972790455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c005040 con 0x7f8fac06b960 2023-11-15T04:32:43.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fb157c700 1 --2- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac06b960 0x7f8fac069e20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8f9c009f80 tx=0x7f8f9c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:43.262 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fa1ffb700 1 -- 172.21.15.190:0/2972790455 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f9c0136c0 con 0x7f8fac06b960 2023-11-15T04:32:43.262 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fac06a310 con 0x7f8fac06b960 2023-11-15T04:32:43.262 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fa1ffb700 1 -- 172.21.15.190:0/2972790455 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f9c0095f0 con 0x7f8fac06b960 2023-11-15T04:32:43.263 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fac06a800 con 0x7f8fac06b960 2023-11-15T04:32:43.263 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.257+0000 7f8fa1ffb700 1 -- 172.21.15.190:0/2972790455 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f9c028440 con 0x7f8fac06b960 2023-11-15T04:32:43.263 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.258+0000 7f8fa1ffb700 1 -- 172.21.15.190:0/2972790455 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f9c025060 con 0x7f8fac06b960 2023-11-15T04:32:43.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.258+0000 7f8fa1ffb700 1 --2- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f9802ff60 0x7f8f98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:43.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.258+0000 7f8fa1ffb700 1 -- 172.21.15.190:0/2972790455 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f9c040dd0 con 0x7f8fac06b960 2023-11-15T04:32:43.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.258+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fac059100 con 0x7f8fac06b960 2023-11-15T04:32:43.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.259+0000 7f8fb0d7b700 1 --2- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f9802ff60 0x7f8f98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:43.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.259+0000 7f8fb0d7b700 1 --2- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f9802ff60 0x7f8f98032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8fa80096a0 tx=0x7f8fa8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:43.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.263+0000 7f8fa1ffb700 1 -- 172.21.15.190:0/2972790455 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f9c014420 con 0x7f8fac06b960 2023-11-15T04:32:43.470 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.469+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8fac045ad0 con 0x7f8fac06b960 2023-11-15T04:32:43.471 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.470+0000 7f8fa1ffb700 1 -- 172.21.15.190:0/2972790455 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f9c024260 con 0x7f8fac06b960 2023-11-15T04:32:43.471 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:43.471 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:43.473 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f9802ff60 msgr2=0x7f8f98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:43.473 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 --2- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f9802ff60 0x7f8f98032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8fa80096a0 tx=0x7f8fa8006b40).stop 2023-11-15T04:32:43.473 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac06b960 msgr2=0x7f8fac069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:43.474 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 --2- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac06b960 0x7f8fac069e20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8f9c009f80 tx=0x7f8f9c009b30).stop 2023-11-15T04:32:43.474 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 shutdown_connections 2023-11-15T04:32:43.474 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 --2- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f9802ff60 0x7f8f98032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:43.474 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 --2- 172.21.15.190:0/2972790455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fac06b960 0x7f8fac069e20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:43.475 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 >> 172.21.15.190:0/2972790455 conn(0x7f8fac0f1c70 msgr2=0x7f8fac0f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:43.475 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 shutdown_connections 2023-11-15T04:32:43.475 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:43.472+0000 7f8fb37e0700 1 -- 172.21.15.190:0/2972790455 wait complete. 2023-11-15T04:32:43.475 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:44.510 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:43.089806+0000 mgr.smithi047.zgysuz (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:44.510 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:43.471516+0000 mon.smithi047 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.190:0/2972790455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:44.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:44.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:45.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.942+0000 7fad41f6e700 1 -- 172.21.15.190:0/3261658227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c0f5410 msgr2=0x7fad3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:45.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.942+0000 7fad41f6e700 1 --2- 172.21.15.190:0/3261658227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c0f5410 0x7fad3c0f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fad2c009fb0 tx=0x7fad2c009b30).stop 2023-11-15T04:32:45.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.942+0000 7fad41f6e700 1 -- 172.21.15.190:0/3261658227 shutdown_connections 2023-11-15T04:32:45.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.942+0000 7fad41f6e700 1 --2- 172.21.15.190:0/3261658227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c0f5410 0x7fad3c0f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:45.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.942+0000 7fad41f6e700 1 -- 172.21.15.190:0/3261658227 >> 172.21.15.190:0/3261658227 conn(0x7fad3c0f0b70 msgr2=0x7fad3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:45.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.942+0000 7fad41f6e700 1 -- 172.21.15.190:0/3261658227 shutdown_connections 2023-11-15T04:32:45.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.942+0000 7fad41f6e700 1 -- 172.21.15.190:0/3261658227 wait complete. 2023-11-15T04:32:45.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.943+0000 7fad41f6e700 1 Processor -- start 2023-11-15T04:32:45.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.943+0000 7fad41f6e700 1 -- start start 2023-11-15T04:32:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.943+0000 7fad41f6e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c106810 0x7fad3c106c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.943+0000 7fad41f6e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c107120 con 0x7fad3c106810 2023-11-15T04:32:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.943+0000 7fad3b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c106810 0x7fad3c106c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.943+0000 7fad3b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c106810 0x7fad3c106c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53466/0 (socket says 172.21.15.190:53466) 2023-11-15T04:32:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.943+0000 7fad3b7fe700 1 -- 172.21.15.190:0/2032298227 learned_addr learned my addr 172.21.15.190:0/2032298227 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.944+0000 7fad3b7fe700 1 -- 172.21.15.190:0/2032298227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c005040 con 0x7fad3c106810 2023-11-15T04:32:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.944+0000 7fad3b7fe700 1 --2- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c106810 0x7fad3c106c30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fad2c009f80 tx=0x7fad2c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.944+0000 7fad23fff700 1 -- 172.21.15.190:0/2032298227 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad2c0136c0 con 0x7fad3c106810 2023-11-15T04:32:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.944+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c107320 con 0x7fad3c106810 2023-11-15T04:32:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.944+0000 7fad23fff700 1 -- 172.21.15.190:0/2032298227 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad2c0095f0 con 0x7fad3c106810 2023-11-15T04:32:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.944+0000 7fad23fff700 1 -- 172.21.15.190:0/2032298227 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad2c014420 con 0x7fad3c106810 2023-11-15T04:32:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.944+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c107e70 con 0x7fad3c106810 2023-11-15T04:32:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.945+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c059100 con 0x7fad3c106810 2023-11-15T04:32:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.945+0000 7fad23fff700 1 -- 172.21.15.190:0/2032298227 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad2c025060 con 0x7fad3c106810 2023-11-15T04:32:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.945+0000 7fad23fff700 1 --2- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fad2402ff60 0x7fad24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.945+0000 7fad23fff700 1 -- 172.21.15.190:0/2032298227 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad2c040d80 con 0x7fad3c106810 2023-11-15T04:32:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.945+0000 7fad3affd700 1 --2- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fad2402ff60 0x7fad24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.946+0000 7fad3affd700 1 --2- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fad2402ff60 0x7fad24032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fad300096a0 tx=0x7fad30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:45.950+0000 7fad23fff700 1 -- 172.21.15.190:0/2032298227 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad2c01cb60 con 0x7fad3c106810 2023-11-15T04:32:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.158+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad3c108630 con 0x7fad3c106810 2023-11-15T04:32:46.160 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.159+0000 7fad23fff700 1 -- 172.21.15.190:0/2032298227 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad2c042070 con 0x7fad3c106810 2023-11-15T04:32:46.161 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:46.161 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:46.163 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fad2402ff60 msgr2=0x7fad24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:46.163 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 --2- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fad2402ff60 0x7fad24032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fad300096a0 tx=0x7fad30006b40).stop 2023-11-15T04:32:46.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c106810 msgr2=0x7fad3c106c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:46.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 --2- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c106810 0x7fad3c106c30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fad2c009f80 tx=0x7fad2c009b30).stop 2023-11-15T04:32:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 shutdown_connections 2023-11-15T04:32:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 --2- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fad2402ff60 0x7fad24032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 --2- 172.21.15.190:0/2032298227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad3c106810 0x7fad3c106c30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 >> 172.21.15.190:0/2032298227 conn(0x7fad3c0f0b70 msgr2=0x7fad3c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 shutdown_connections 2023-11-15T04:32:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:46.160+0000 7fad41f6e700 1 -- 172.21.15.190:0/2032298227 wait complete. 2023-11-15T04:32:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:45.090091+0000 mgr.smithi047.zgysuz (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:47.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:46.160200+0000 mon.smithi047 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.190:0/2032298227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:47.540 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:47.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:48.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:48 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:47.090378+0000 mgr.smithi047.zgysuz (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:48.722 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.718+0000 7f6b43a6c700 1 -- 172.21.15.190:0/296090092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 msgr2=0x7f6b3c0f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:48.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.718+0000 7f6b43a6c700 1 --2- 172.21.15.190:0/296090092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 0x7f6b3c0f9170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f6b2c009fb0 tx=0x7f6b2c009b30).stop 2023-11-15T04:32:48.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.718+0000 7f6b43a6c700 1 -- 172.21.15.190:0/296090092 shutdown_connections 2023-11-15T04:32:48.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.718+0000 7f6b43a6c700 1 --2- 172.21.15.190:0/296090092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 0x7f6b3c0f9170 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:48.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.718+0000 7f6b43a6c700 1 -- 172.21.15.190:0/296090092 >> 172.21.15.190:0/296090092 conn(0x7f6b3c0f0b70 msgr2=0x7f6b3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:48.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.718+0000 7f6b43a6c700 1 -- 172.21.15.190:0/296090092 shutdown_connections 2023-11-15T04:32:48.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.718+0000 7f6b43a6c700 1 -- 172.21.15.190:0/296090092 wait complete. 2023-11-15T04:32:48.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.718+0000 7f6b43a6c700 1 Processor -- start 2023-11-15T04:32:48.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.718+0000 7f6b43a6c700 1 -- start start 2023-11-15T04:32:48.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b43a6c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 0x7f6b3c1066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:48.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b43a6c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c106bb0 con 0x7f6b3c060200 2023-11-15T04:32:48.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b41808700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 0x7f6b3c1066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:48.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b41808700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 0x7f6b3c1066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:53484/0 (socket says 172.21.15.190:53484) 2023-11-15T04:32:48.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b41808700 1 -- 172.21.15.190:0/3931581060 learned_addr learned my addr 172.21.15.190:0/3931581060 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:48.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b41808700 1 -- 172.21.15.190:0/3931581060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c005040 con 0x7f6b3c060200 2023-11-15T04:32:48.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b41808700 1 --2- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 0x7f6b3c1066c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6b2c013510 tx=0x7f6b2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:48.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b327fc700 1 -- 172.21.15.190:0/3931581060 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b2c009320 con 0x7f6b3c060200 2023-11-15T04:32:48.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c106db0 con 0x7f6b3c060200 2023-11-15T04:32:48.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b327fc700 1 -- 172.21.15.190:0/3931581060 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b2c013f00 con 0x7f6b3c060200 2023-11-15T04:32:48.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b327fc700 1 -- 172.21.15.190:0/3931581060 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b2c014dc0 con 0x7f6b3c060200 2023-11-15T04:32:48.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.719+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c107240 con 0x7f6b3c060200 2023-11-15T04:32:48.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.720+0000 7f6b327fc700 1 -- 172.21.15.190:0/3931581060 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b2c0095f0 con 0x7f6b3c060200 2023-11-15T04:32:48.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.720+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c045ad0 con 0x7f6b3c060200 2023-11-15T04:32:48.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.720+0000 7f6b327fc700 1 --2- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6b2802ffb0 0x7f6b28032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:48.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.720+0000 7f6b327fc700 1 -- 172.21.15.190:0/3931581060 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b2c041050 con 0x7f6b3c060200 2023-11-15T04:32:48.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.721+0000 7f6b41007700 1 --2- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6b2802ffb0 0x7f6b28032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:48.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.721+0000 7f6b41007700 1 --2- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6b2802ffb0 0x7f6b28032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6b380096a0 tx=0x7f6b38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:48.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.725+0000 7f6b327fc700 1 -- 172.21.15.190:0/3931581060 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b2c014420 con 0x7f6b3c060200 2023-11-15T04:32:48.932 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.931+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b3c1080f0 con 0x7f6b3c060200 2023-11-15T04:32:48.933 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.932+0000 7f6b327fc700 1 -- 172.21.15.190:0/3931581060 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b2c010230 con 0x7f6b3c060200 2023-11-15T04:32:48.934 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:48.934 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:48.935 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.933+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6b2802ffb0 msgr2=0x7f6b28032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:48.935 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.933+0000 7f6b43a6c700 1 --2- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6b2802ffb0 0x7f6b28032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6b380096a0 tx=0x7f6b38006b40).stop 2023-11-15T04:32:48.935 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.933+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 msgr2=0x7f6b3c1066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:48.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.933+0000 7f6b43a6c700 1 --2- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 0x7f6b3c1066c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6b2c013510 tx=0x7f6b2c00b040).stop 2023-11-15T04:32:48.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.933+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 shutdown_connections 2023-11-15T04:32:48.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.933+0000 7f6b43a6c700 1 --2- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6b2802ffb0 0x7f6b28032470 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:48.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.933+0000 7f6b43a6c700 1 --2- 172.21.15.190:0/3931581060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b3c060200 0x7f6b3c1066c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:48.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.933+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 >> 172.21.15.190:0/3931581060 conn(0x7f6b3c0f0b70 msgr2=0x7f6b3c0f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:48.937 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.934+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 shutdown_connections 2023-11-15T04:32:48.937 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:48.934+0000 7f6b43a6c700 1 -- 172.21.15.190:0/3931581060 wait complete. 2023-11-15T04:32:48.937 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:49.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:49 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:48.933477+0000 mon.smithi047 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.190:0/3931581060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:50.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:50.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:50.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:50 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:49.090657+0000 mgr.smithi047.zgysuz (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:51.461 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.459+0000 7feea9347700 1 -- 172.21.15.190:0/1439798436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 msgr2=0x7feea40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:51.462 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.459+0000 7feea9347700 1 --2- 172.21.15.190:0/1439798436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 0x7feea40f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fee94009fb0 tx=0x7fee94009b30).stop 2023-11-15T04:32:51.462 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.459+0000 7feea9347700 1 -- 172.21.15.190:0/1439798436 shutdown_connections 2023-11-15T04:32:51.462 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.459+0000 7feea9347700 1 --2- 172.21.15.190:0/1439798436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 0x7feea40f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:51.462 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.459+0000 7feea9347700 1 -- 172.21.15.190:0/1439798436 >> 172.21.15.190:0/1439798436 conn(0x7feea40f0b70 msgr2=0x7feea40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:51.462 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.459+0000 7feea9347700 1 -- 172.21.15.190:0/1439798436 shutdown_connections 2023-11-15T04:32:51.463 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.459+0000 7feea9347700 1 -- 172.21.15.190:0/1439798436 wait complete. 2023-11-15T04:32:51.463 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.460+0000 7feea9347700 1 Processor -- start 2023-11-15T04:32:51.463 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.460+0000 7feea9347700 1 -- start start 2023-11-15T04:32:51.463 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.460+0000 7feea9347700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 0x7feea40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:51.463 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.460+0000 7feea9347700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea40fb820 con 0x7feea40f5410 2023-11-15T04:32:51.464 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.460+0000 7feea2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 0x7feea40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:51.464 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.460+0000 7feea2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 0x7feea40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:42594/0 (socket says 172.21.15.190:42594) 2023-11-15T04:32:51.464 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.460+0000 7feea2ffd700 1 -- 172.21.15.190:0/1708548945 learned_addr learned my addr 172.21.15.190:0/1708548945 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:51.464 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.461+0000 7feea2ffd700 1 -- 172.21.15.190:0/1708548945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee94005040 con 0x7feea40f5410 2023-11-15T04:32:51.465 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.461+0000 7feea2ffd700 1 --2- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 0x7feea40fd3a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fee94013510 tx=0x7fee9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:51.465 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.461+0000 7fee9b7fe700 1 -- 172.21.15.190:0/1708548945 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee94009320 con 0x7feea40f5410 2023-11-15T04:32:51.468 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.461+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea40fb9c0 con 0x7feea40f5410 2023-11-15T04:32:51.468 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.461+0000 7fee9b7fe700 1 -- 172.21.15.190:0/1708548945 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee94013f00 con 0x7feea40f5410 2023-11-15T04:32:51.469 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.461+0000 7fee9b7fe700 1 -- 172.21.15.190:0/1708548945 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee94014d90 con 0x7feea40f5410 2023-11-15T04:32:51.469 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.461+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea40fbe50 con 0x7feea40f5410 2023-11-15T04:32:51.469 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.462+0000 7fee9b7fe700 1 -- 172.21.15.190:0/1708548945 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee940095f0 con 0x7feea40f5410 2023-11-15T04:32:51.470 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.462+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea4059100 con 0x7feea40f5410 2023-11-15T04:32:51.470 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.462+0000 7fee9b7fe700 1 --2- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fee9002ff60 0x7fee90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:51.470 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.462+0000 7fee9b7fe700 1 -- 172.21.15.190:0/1708548945 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee94040fc0 con 0x7feea40f5410 2023-11-15T04:32:51.470 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.462+0000 7feea27fc700 1 --2- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fee9002ff60 0x7fee90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:51.470 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.462+0000 7feea27fc700 1 --2- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fee9002ff60 0x7fee90032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fee8c0096a0 tx=0x7fee8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:51.471 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.467+0000 7fee9b7fe700 1 -- 172.21.15.190:0/1708548945 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee94014420 con 0x7feea40f5410 2023-11-15T04:32:51.673 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.672+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feea4045ad0 con 0x7feea40f5410 2023-11-15T04:32:51.674 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.673+0000 7fee9b7fe700 1 -- 172.21.15.190:0/1708548945 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee94010230 con 0x7feea40f5410 2023-11-15T04:32:51.674 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:51.674 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:51.675 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fee9002ff60 msgr2=0x7fee90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:51.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 --2- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fee9002ff60 0x7fee90032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fee8c0096a0 tx=0x7fee8c006b40).stop 2023-11-15T04:32:51.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 msgr2=0x7feea40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:51.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 --2- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 0x7feea40fd3a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fee94013510 tx=0x7fee9400b040).stop 2023-11-15T04:32:51.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 shutdown_connections 2023-11-15T04:32:51.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 --2- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fee9002ff60 0x7fee90032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 --2- 172.21.15.190:0/1708548945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feea40f5410 0x7feea40fd3a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 >> 172.21.15.190:0/1708548945 conn(0x7feea40f0b70 msgr2=0x7feea405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 shutdown_connections 2023-11-15T04:32:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:51.675+0000 7feea9347700 1 -- 172.21.15.190:0/1708548945 wait complete. 2023-11-15T04:32:51.678 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:51.090958+0000 mgr.smithi047.zgysuz (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:52.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:51.674307+0000 mon.smithi047 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.190:0/1708548945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:53.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:53.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:54.240 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 -- 172.21.15.190:0/1295059957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 msgr2=0x7fb6100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:54.240 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 --2- 172.21.15.190:0/1295059957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 0x7fb6100f55c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fb600009fb0 tx=0x7fb600009b30).stop 2023-11-15T04:32:54.240 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 -- 172.21.15.190:0/1295059957 shutdown_connections 2023-11-15T04:32:54.240 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 --2- 172.21.15.190:0/1295059957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 0x7fb6100f55c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:54.241 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 -- 172.21.15.190:0/1295059957 >> 172.21.15.190:0/1295059957 conn(0x7fb6100f09a0 msgr2=0x7fb6100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:54.241 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 -- 172.21.15.190:0/1295059957 shutdown_connections 2023-11-15T04:32:54.241 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 -- 172.21.15.190:0/1295059957 wait complete. 2023-11-15T04:32:54.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 Processor -- start 2023-11-15T04:32:54.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 -- start start 2023-11-15T04:32:54.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 0x7fb6101091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:54.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.239+0000 7fb60ffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb610060220 con 0x7fb6100f51a0 2023-11-15T04:32:54.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.240+0000 7fb60effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 0x7fb6101091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:54.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.240+0000 7fb60effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 0x7fb6101091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:42618/0 (socket says 172.21.15.190:42618) 2023-11-15T04:32:54.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.240+0000 7fb60effd700 1 -- 172.21.15.190:0/3253046144 learned_addr learned my addr 172.21.15.190:0/3253046144 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:54.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.240+0000 7fb60effd700 1 -- 172.21.15.190:0/3253046144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb600005040 con 0x7fb6100f51a0 2023-11-15T04:32:54.245 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.240+0000 7fb60effd700 1 --2- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 0x7fb6101091d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fb600009830 tx=0x7fb60000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:54.245 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.240+0000 7fb5f77fe700 1 -- 172.21.15.190:0/3253046144 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb600009320 con 0x7fb6100f51a0 2023-11-15T04:32:54.245 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.241+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb610060420 con 0x7fb6100f51a0 2023-11-15T04:32:54.245 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.241+0000 7fb5f77fe700 1 -- 172.21.15.190:0/3253046144 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6000136c0 con 0x7fb6100f51a0 2023-11-15T04:32:54.245 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.241+0000 7fb5f77fe700 1 -- 172.21.15.190:0/3253046144 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60001ee90 con 0x7fb6100f51a0 2023-11-15T04:32:54.246 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.241+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb610109880 con 0x7fb6100f51a0 2023-11-15T04:32:54.246 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.241+0000 7fb5f77fe700 1 -- 172.21.15.190:0/3253046144 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb60001e420 con 0x7fb6100f51a0 2023-11-15T04:32:54.246 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.241+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb610059100 con 0x7fb6100f51a0 2023-11-15T04:32:54.246 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.242+0000 7fb5f77fe700 1 --2- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb5f802ff10 0x7fb5f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:54.246 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.242+0000 7fb5f77fe700 1 -- 172.21.15.190:0/3253046144 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb600040f60 con 0x7fb6100f51a0 2023-11-15T04:32:54.247 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.242+0000 7fb60e7fc700 1 --2- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb5f802ff10 0x7fb5f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:54.247 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.242+0000 7fb60e7fc700 1 --2- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb5f802ff10 0x7fb5f80323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb6040096a0 tx=0x7fb604006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:54.247 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.247+0000 7fb5f77fe700 1 -- 172.21.15.190:0/3253046144 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb60001c790 con 0x7fb6100f51a0 2023-11-15T04:32:54.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.451+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6100fabe0 con 0x7fb6100f51a0 2023-11-15T04:32:54.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.451+0000 7fb5f77fe700 1 -- 172.21.15.190:0/3253046144 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb60002c2d0 con 0x7fb6100f51a0 2023-11-15T04:32:54.453 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:54.453 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:54.454 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb5f802ff10 msgr2=0x7fb5f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:54.454 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 --2- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb5f802ff10 0x7fb5f80323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb6040096a0 tx=0x7fb604006b40).stop 2023-11-15T04:32:54.454 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 msgr2=0x7fb6101091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:54.454 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 --2- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 0x7fb6101091d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fb600009830 tx=0x7fb60000b040).stop 2023-11-15T04:32:54.455 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 shutdown_connections 2023-11-15T04:32:54.455 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 --2- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb5f802ff10 0x7fb5f80323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:54.455 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 --2- 172.21.15.190:0/3253046144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6100f51a0 0x7fb6101091d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:54.455 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 >> 172.21.15.190:0/3253046144 conn(0x7fb6100f09a0 msgr2=0x7fb6100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:54.455 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 shutdown_connections 2023-11-15T04:32:54.456 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:54.453+0000 7fb60ffff700 1 -- 172.21.15.190:0/3253046144 wait complete. 2023-11-15T04:32:54.456 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:53.091259+0000 mgr.smithi047.zgysuz (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:55.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:55 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:54.453069+0000 mon.smithi047 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.190:0/3253046144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:55.842 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:55.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:55.091572+0000 mgr.smithi047.zgysuz (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:56.991 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.989+0000 7fb4d5171700 1 -- 172.21.15.190:0/2444061296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 msgr2=0x7fb4d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:56.992 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.989+0000 7fb4d5171700 1 --2- 172.21.15.190:0/2444061296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 0x7fb4d00f55c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fb4c0009fb0 tx=0x7fb4c0009b30).stop 2023-11-15T04:32:56.992 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.989+0000 7fb4d5171700 1 -- 172.21.15.190:0/2444061296 shutdown_connections 2023-11-15T04:32:56.992 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.989+0000 7fb4d5171700 1 --2- 172.21.15.190:0/2444061296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 0x7fb4d00f55c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:56.992 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.989+0000 7fb4d5171700 1 -- 172.21.15.190:0/2444061296 >> 172.21.15.190:0/2444061296 conn(0x7fb4d00f09a0 msgr2=0x7fb4d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:56.992 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.989+0000 7fb4d5171700 1 -- 172.21.15.190:0/2444061296 shutdown_connections 2023-11-15T04:32:56.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.989+0000 7fb4d5171700 1 -- 172.21.15.190:0/2444061296 wait complete. 2023-11-15T04:32:56.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.989+0000 7fb4d5171700 1 Processor -- start 2023-11-15T04:32:56.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.989+0000 7fb4d5171700 1 -- start start 2023-11-15T04:32:56.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4d5171700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 0x7fb4d01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:56.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4d5171700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d0060240 con 0x7fb4d00f51a0 2023-11-15T04:32:56.994 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4cffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 0x7fb4d01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:56.994 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4cffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 0x7fb4d01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:42644/0 (socket says 172.21.15.190:42644) 2023-11-15T04:32:56.994 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4cffff700 1 -- 172.21.15.190:0/966401936 learned_addr learned my addr 172.21.15.190:0/966401936 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:56.994 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4cffff700 1 -- 172.21.15.190:0/966401936 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c0005040 con 0x7fb4d00f51a0 2023-11-15T04:32:56.994 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4cffff700 1 --2- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 0x7fb4d01091b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fb4c0013510 tx=0x7fb4c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:56.995 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4ccff9700 1 -- 172.21.15.190:0/966401936 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4c0009320 con 0x7fb4d00f51a0 2023-11-15T04:32:56.995 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d0060440 con 0x7fb4d00f51a0 2023-11-15T04:32:56.995 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4ccff9700 1 -- 172.21.15.190:0/966401936 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4c0013f00 con 0x7fb4d00f51a0 2023-11-15T04:32:56.995 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4ccff9700 1 -- 172.21.15.190:0/966401936 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4c0014dc0 con 0x7fb4d00f51a0 2023-11-15T04:32:56.995 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.990+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d0109860 con 0x7fb4d00f51a0 2023-11-15T04:32:56.996 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.991+0000 7fb4ccff9700 1 -- 172.21.15.190:0/966401936 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4c0014420 con 0x7fb4d00f51a0 2023-11-15T04:32:56.997 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.991+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d0045ad0 con 0x7fb4d00f51a0 2023-11-15T04:32:56.997 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.991+0000 7fb4ccff9700 1 --2- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4b802ff60 0x7fb4b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:56.997 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.991+0000 7fb4ccff9700 1 -- 172.21.15.190:0/966401936 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4c0041020 con 0x7fb4d00f51a0 2023-11-15T04:32:56.997 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.992+0000 7fb4cf7fe700 1 --2- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4b802ff60 0x7fb4b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:56.997 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.992+0000 7fb4cf7fe700 1 --2- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4b802ff60 0x7fb4b8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb4c40096a0 tx=0x7fb4c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:56.998 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:56.996+0000 7fb4ccff9700 1 -- 172.21.15.190:0/966401936 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4c00095f0 con 0x7fb4d00f51a0 2023-11-15T04:32:57.203 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.202+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4d010a070 con 0x7fb4d00f51a0 2023-11-15T04:32:57.204 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.203+0000 7fb4ccff9700 1 -- 172.21.15.190:0/966401936 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4c001cb10 con 0x7fb4d00f51a0 2023-11-15T04:32:57.205 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:57.205 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:57.206 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.204+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4b802ff60 msgr2=0x7fb4b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:57.206 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.204+0000 7fb4d5171700 1 --2- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4b802ff60 0x7fb4b8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb4c40096a0 tx=0x7fb4c4006b40).stop 2023-11-15T04:32:57.206 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.204+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 msgr2=0x7fb4d01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:57.206 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.204+0000 7fb4d5171700 1 --2- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 0x7fb4d01091b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fb4c0013510 tx=0x7fb4c000b040).stop 2023-11-15T04:32:57.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.204+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 shutdown_connections 2023-11-15T04:32:57.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.204+0000 7fb4d5171700 1 --2- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4b802ff60 0x7fb4b8032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:57.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.204+0000 7fb4d5171700 1 --2- 172.21.15.190:0/966401936 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4d00f51a0 0x7fb4d01091b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:57.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.204+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 >> 172.21.15.190:0/966401936 conn(0x7fb4d00f09a0 msgr2=0x7fb4d00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:57.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.205+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 shutdown_connections 2023-11-15T04:32:57.207 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:57.205+0000 7fb4d5171700 1 -- 172.21.15.190:0/966401936 wait complete. 2023-11-15T04:32:57.208 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:32:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:57.091916+0000 mgr.smithi047.zgysuz (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:32:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:32:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:57.204190+0000 mon.smithi047 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.190:0/966401936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:32:58.613 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:32:58.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:32:59.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.749+0000 7fc8716dc700 1 -- 172.21.15.190:0/2198667310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 msgr2=0x7fc86c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:59.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.749+0000 7fc8716dc700 1 --2- 172.21.15.190:0/2198667310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 0x7fc86c0f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc860009fb0 tx=0x7fc860009b30).stop 2023-11-15T04:32:59.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.749+0000 7fc8716dc700 1 -- 172.21.15.190:0/2198667310 shutdown_connections 2023-11-15T04:32:59.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.749+0000 7fc8716dc700 1 --2- 172.21.15.190:0/2198667310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 0x7fc86c0f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:59.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.749+0000 7fc8716dc700 1 -- 172.21.15.190:0/2198667310 >> 172.21.15.190:0/2198667310 conn(0x7fc86c0f0b70 msgr2=0x7fc86c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:59.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.749+0000 7fc8716dc700 1 -- 172.21.15.190:0/2198667310 shutdown_connections 2023-11-15T04:32:59.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.749+0000 7fc8716dc700 1 -- 172.21.15.190:0/2198667310 wait complete. 2023-11-15T04:32:59.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.750+0000 7fc8716dc700 1 Processor -- start 2023-11-15T04:32:59.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.750+0000 7fc8716dc700 1 -- start start 2023-11-15T04:32:59.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.750+0000 7fc8716dc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 0x7fc86c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:59.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.750+0000 7fc8716dc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c107240 con 0x7fc86c0f5410 2023-11-15T04:32:59.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.750+0000 7fc86affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 0x7fc86c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:59.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.750+0000 7fc86affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 0x7fc86c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:42654/0 (socket says 172.21.15.190:42654) 2023-11-15T04:32:59.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.750+0000 7fc86affd700 1 -- 172.21.15.190:0/3422578892 learned_addr learned my addr 172.21.15.190:0/3422578892 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:32:59.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.750+0000 7fc86affd700 1 -- 172.21.15.190:0/3422578892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc860005040 con 0x7fc86c0f5410 2023-11-15T04:32:59.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.751+0000 7fc86affd700 1 --2- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 0x7fc86c10b850 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fc860013510 tx=0x7fc86000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:59.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.751+0000 7fc8537fe700 1 -- 172.21.15.190:0/3422578892 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc860009320 con 0x7fc86c0f5410 2023-11-15T04:32:59.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.751+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c060220 con 0x7fc86c0f5410 2023-11-15T04:32:59.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.751+0000 7fc8537fe700 1 -- 172.21.15.190:0/3422578892 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc860013f00 con 0x7fc86c0f5410 2023-11-15T04:32:59.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.751+0000 7fc8537fe700 1 -- 172.21.15.190:0/3422578892 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc860014d90 con 0x7fc86c0f5410 2023-11-15T04:32:59.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.751+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c10bd40 con 0x7fc86c0f5410 2023-11-15T04:32:59.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.752+0000 7fc8537fe700 1 -- 172.21.15.190:0/3422578892 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8600263a0 con 0x7fc86c0f5410 2023-11-15T04:32:59.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.752+0000 7fc8537fe700 1 --2- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc85402ff60 0x7fc854032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:32:59.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.752+0000 7fc8537fe700 1 -- 172.21.15.190:0/3422578892 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc860040fb0 con 0x7fc86c0f5410 2023-11-15T04:32:59.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.752+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c045ad0 con 0x7fc86c0f5410 2023-11-15T04:32:59.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.752+0000 7fc86a7fc700 1 --2- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc85402ff60 0x7fc854032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:32:59.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.752+0000 7fc86a7fc700 1 --2- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc85402ff60 0x7fc854032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc85c0096a0 tx=0x7fc85c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:32:59.758 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.757+0000 7fc8537fe700 1 -- 172.21.15.190:0/3422578892 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc860012920 con 0x7fc86c0f5410 2023-11-15T04:32:59.963 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.962+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc86c0fae60 con 0x7fc86c0f5410 2023-11-15T04:32:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.963+0000 7fc8537fe700 1 -- 172.21.15.190:0/3422578892 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc86c0fae60 con 0x7fc86c0f5410 2023-11-15T04:32:59.964 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:32:59.964 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:32:59.966 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc85402ff60 msgr2=0x7fc854032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:59.966 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 --2- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc85402ff60 0x7fc854032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc85c0096a0 tx=0x7fc85c006b40).stop 2023-11-15T04:32:59.966 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 msgr2=0x7fc86c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:32:59.966 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 --2- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 0x7fc86c10b850 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fc860013510 tx=0x7fc86000b040).stop 2023-11-15T04:32:59.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 shutdown_connections 2023-11-15T04:32:59.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 --2- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc85402ff60 0x7fc854032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:59.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 --2- 172.21.15.190:0/3422578892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc86c0f5410 0x7fc86c10b850 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:32:59.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 >> 172.21.15.190:0/3422578892 conn(0x7fc86c0f0b70 msgr2=0x7fc86c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:32:59.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 shutdown_connections 2023-11-15T04:32:59.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:32:59.965+0000 7fc8716dc700 1 -- 172.21.15.190:0/3422578892 wait complete. 2023-11-15T04:32:59.968 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:32:59.092253+0000 mgr.smithi047.zgysuz (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:32:59.964478+0000 mon.smithi047 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.190:0/3422578892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:01.370 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:01.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:01.092556+0000 mgr.smithi047.zgysuz (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:02.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.537+0000 7f03333cb700 1 -- 172.21.15.190:0/107707836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 msgr2=0x7f032c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:02.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.537+0000 7f03333cb700 1 --2- 172.21.15.190:0/107707836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 0x7f032c0f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f0328009fb0 tx=0x7f0328009b30).stop 2023-11-15T04:33:02.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.537+0000 7f03333cb700 1 -- 172.21.15.190:0/107707836 shutdown_connections 2023-11-15T04:33:02.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.537+0000 7f03333cb700 1 --2- 172.21.15.190:0/107707836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 0x7f032c0f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:02.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.537+0000 7f03333cb700 1 -- 172.21.15.190:0/107707836 >> 172.21.15.190:0/107707836 conn(0x7f032c0f0b70 msgr2=0x7f032c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:02.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.537+0000 7f03333cb700 1 -- 172.21.15.190:0/107707836 shutdown_connections 2023-11-15T04:33:02.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.537+0000 7f03333cb700 1 -- 172.21.15.190:0/107707836 wait complete. 2023-11-15T04:33:02.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.538+0000 7f03333cb700 1 Processor -- start 2023-11-15T04:33:02.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.538+0000 7f03333cb700 1 -- start start 2023-11-15T04:33:02.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.538+0000 7f03333cb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 0x7f032c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:02.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.538+0000 7f03333cb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c105b20 con 0x7f032c0f5410 2023-11-15T04:33:02.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.538+0000 7f0331167700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 0x7f032c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:02.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.538+0000 7f0331167700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 0x7f032c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34684/0 (socket says 172.21.15.190:34684) 2023-11-15T04:33:02.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.538+0000 7f0331167700 1 -- 172.21.15.190:0/694199651 learned_addr learned my addr 172.21.15.190:0/694199651 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:02.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.539+0000 7f0331167700 1 -- 172.21.15.190:0/694199651 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328005040 con 0x7f032c0f5410 2023-11-15T04:33:02.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.539+0000 7f0331167700 1 --2- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 0x7f032c105630 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0328014000 tx=0x7f032800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:02.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.539+0000 7f0321ffb700 1 -- 172.21.15.190:0/694199651 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03280095f0 con 0x7f032c0f5410 2023-11-15T04:33:02.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.539+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c105d20 con 0x7f032c0f5410 2023-11-15T04:33:02.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.539+0000 7f0321ffb700 1 -- 172.21.15.190:0/694199651 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0328007e70 con 0x7f032c0f5410 2023-11-15T04:33:02.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.539+0000 7f0321ffb700 1 -- 172.21.15.190:0/694199651 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0328014ea0 con 0x7f032c0f5410 2023-11-15T04:33:02.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.539+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c1061b0 con 0x7f032c0f5410 2023-11-15T04:33:02.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.540+0000 7f0321ffb700 1 -- 172.21.15.190:0/694199651 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0328025070 con 0x7f032c0f5410 2023-11-15T04:33:02.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.540+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032c045ad0 con 0x7f032c0f5410 2023-11-15T04:33:02.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.540+0000 7f0321ffb700 1 --2- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f031802ff60 0x7f0318032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:02.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.540+0000 7f0321ffb700 1 -- 172.21.15.190:0/694199651 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0328040920 con 0x7f032c0f5410 2023-11-15T04:33:02.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.540+0000 7f0330966700 1 --2- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f031802ff60 0x7f0318032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:02.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.541+0000 7f0330966700 1 --2- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f031802ff60 0x7f0318032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f031c0096a0 tx=0x7f031c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:02.547 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.545+0000 7f0321ffb700 1 -- 172.21.15.190:0/694199651 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f032800f4e0 con 0x7f032c0f5410 2023-11-15T04:33:02.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.751+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f032c0fae60 con 0x7f032c0f5410 2023-11-15T04:33:02.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.752+0000 7f0321ffb700 1 -- 172.21.15.190:0/694199651 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f032800f300 con 0x7f032c0f5410 2023-11-15T04:33:02.753 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:02.753 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:02.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f031802ff60 msgr2=0x7f0318032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:02.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 --2- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f031802ff60 0x7f0318032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f031c0096a0 tx=0x7f031c006b40).stop 2023-11-15T04:33:02.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 msgr2=0x7f032c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:02.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 --2- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 0x7f032c105630 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0328014000 tx=0x7f032800b040).stop 2023-11-15T04:33:02.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 shutdown_connections 2023-11-15T04:33:02.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 --2- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f031802ff60 0x7f0318032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:02.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 --2- 172.21.15.190:0/694199651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f032c0f5410 0x7f032c105630 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:02.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 >> 172.21.15.190:0/694199651 conn(0x7f032c0f0b70 msgr2=0x7f032c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:02.756 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 shutdown_connections 2023-11-15T04:33:02.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:02.753+0000 7f03333cb700 1 -- 172.21.15.190:0/694199651 wait complete. 2023-11-15T04:33:02.757 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:03.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:03 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:02.753184+0000 mon.smithi047 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.190:0/694199651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:04.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:04.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:03.092889+0000 mgr.smithi047.zgysuz (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:05.273 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.270+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/3460843063 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc480f6510 msgr2=0x7fdc480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:05.273 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.270+0000 7fdc4e2fd700 1 --2- 172.21.15.190:0/3460843063 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc480f6510 0x7fdc480f6930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fdc38009fb0 tx=0x7fdc38009b30).stop 2023-11-15T04:33:05.273 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.270+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/3460843063 shutdown_connections 2023-11-15T04:33:05.274 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.270+0000 7fdc4e2fd700 1 --2- 172.21.15.190:0/3460843063 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc480f6510 0x7fdc480f6930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:05.274 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.270+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/3460843063 >> 172.21.15.190:0/3460843063 conn(0x7fdc480f1c70 msgr2=0x7fdc480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:05.274 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.271+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/3460843063 shutdown_connections 2023-11-15T04:33:05.274 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.271+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/3460843063 wait complete. 2023-11-15T04:33:05.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.271+0000 7fdc4e2fd700 1 Processor -- start 2023-11-15T04:33:05.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.271+0000 7fdc4e2fd700 1 -- start start 2023-11-15T04:33:05.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.271+0000 7fdc4e2fd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc48106840 0x7fdc48106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:05.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.271+0000 7fdc4e2fd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48107150 con 0x7fdc48106840 2023-11-15T04:33:05.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.272+0000 7fdc47fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc48106840 0x7fdc48106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:05.275 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.272+0000 7fdc47fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc48106840 0x7fdc48106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34702/0 (socket says 172.21.15.190:34702) 2023-11-15T04:33:05.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.272+0000 7fdc47fff700 1 -- 172.21.15.190:0/1277485471 learned_addr learned my addr 172.21.15.190:0/1277485471 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:05.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.272+0000 7fdc47fff700 1 -- 172.21.15.190:0/1277485471 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc38005040 con 0x7fdc48106840 2023-11-15T04:33:05.276 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.272+0000 7fdc47fff700 1 --2- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc48106840 0x7fdc48106c60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fdc38009f80 tx=0x7fdc38009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:05.277 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.273+0000 7fdc457fa700 1 -- 172.21.15.190:0/1277485471 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc380136c0 con 0x7fdc48106840 2023-11-15T04:33:05.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.273+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc48107350 con 0x7fdc48106840 2023-11-15T04:33:05.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.273+0000 7fdc457fa700 1 -- 172.21.15.190:0/1277485471 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc38013e70 con 0x7fdc48106840 2023-11-15T04:33:05.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.273+0000 7fdc457fa700 1 -- 172.21.15.190:0/1277485471 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc38014420 con 0x7fdc48106840 2023-11-15T04:33:05.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.273+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc48107ea0 con 0x7fdc48106840 2023-11-15T04:33:05.278 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.273+0000 7fdc457fa700 1 -- 172.21.15.190:0/1277485471 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc38025060 con 0x7fdc48106840 2023-11-15T04:33:05.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.273+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc48059100 con 0x7fdc48106840 2023-11-15T04:33:05.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.274+0000 7fdc457fa700 1 --2- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc3402ff60 0x7fdc34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:05.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.274+0000 7fdc457fa700 1 -- 172.21.15.190:0/1277485471 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc38040e50 con 0x7fdc48106840 2023-11-15T04:33:05.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.274+0000 7fdc3ffff700 1 --2- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc3402ff60 0x7fdc34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:05.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.274+0000 7fdc3ffff700 1 --2- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc3402ff60 0x7fdc34032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fdc300096a0 tx=0x7fdc30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:05.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.278+0000 7fdc457fa700 1 -- 172.21.15.190:0/1277485471 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc3801cb30 con 0x7fdc48106840 2023-11-15T04:33:05.488 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.487+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc48045ad0 con 0x7fdc48106840 2023-11-15T04:33:05.489 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.488+0000 7fdc457fa700 1 -- 172.21.15.190:0/1277485471 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc3802d2d0 con 0x7fdc48106840 2023-11-15T04:33:05.489 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:05.490 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:05.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc3402ff60 msgr2=0x7fdc34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:05.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 --2- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc3402ff60 0x7fdc34032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fdc300096a0 tx=0x7fdc30006b40).stop 2023-11-15T04:33:05.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc48106840 msgr2=0x7fdc48106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:05.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 --2- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc48106840 0x7fdc48106c60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fdc38009f80 tx=0x7fdc38009b30).stop 2023-11-15T04:33:05.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 shutdown_connections 2023-11-15T04:33:05.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 --2- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc3402ff60 0x7fdc34032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:05.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 --2- 172.21.15.190:0/1277485471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc48106840 0x7fdc48106c60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:05.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 >> 172.21.15.190:0/1277485471 conn(0x7fdc480f1c70 msgr2=0x7fdc480f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:05.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 shutdown_connections 2023-11-15T04:33:05.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:05.490+0000 7fdc4e2fd700 1 -- 172.21.15.190:0/1277485471 wait complete. 2023-11-15T04:33:05.493 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:05.093227+0000 mgr.smithi047.zgysuz (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:05.217206+0000 mon.smithi047 (mon.0) 200 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:33:06.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:05.219565+0000 mon.smithi047 (mon.0) 201 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:33:06.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:05.489682+0000 mon.smithi047 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.190:0/1277485471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:06.876 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:06.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:08.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:07.093629+0000 mgr.smithi047.zgysuz (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:08.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.006+0000 7f01261c5700 1 -- 172.21.15.190:0/1900906778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 msgr2=0x7f01200f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:08.009 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.006+0000 7f01261c5700 1 --2- 172.21.15.190:0/1900906778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 0x7f01200f9150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f0108009fb0 tx=0x7f0108009b30).stop 2023-11-15T04:33:08.009 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.006+0000 7f01261c5700 1 -- 172.21.15.190:0/1900906778 shutdown_connections 2023-11-15T04:33:08.009 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.006+0000 7f01261c5700 1 --2- 172.21.15.190:0/1900906778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 0x7f01200f9150 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:08.009 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.006+0000 7f01261c5700 1 -- 172.21.15.190:0/1900906778 >> 172.21.15.190:0/1900906778 conn(0x7f01200f0b70 msgr2=0x7f01200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:08.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.006+0000 7f01261c5700 1 -- 172.21.15.190:0/1900906778 shutdown_connections 2023-11-15T04:33:08.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.006+0000 7f01261c5700 1 -- 172.21.15.190:0/1900906778 wait complete. 2023-11-15T04:33:08.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.007+0000 7f01261c5700 1 Processor -- start 2023-11-15T04:33:08.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.007+0000 7f01261c5700 1 -- start start 2023-11-15T04:33:08.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.007+0000 7f01261c5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 0x7f01200f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:08.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.007+0000 7f01261c5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01200f5050 con 0x7f0120060220 2023-11-15T04:33:08.011 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.007+0000 7f011f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 0x7f01200f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:08.011 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.007+0000 7f011f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 0x7f01200f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34726/0 (socket says 172.21.15.190:34726) 2023-11-15T04:33:08.011 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.007+0000 7f011f7fe700 1 -- 172.21.15.190:0/400713013 learned_addr learned my addr 172.21.15.190:0/400713013 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:08.011 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.007+0000 7f011f7fe700 1 -- 172.21.15.190:0/400713013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0108005040 con 0x7f0120060220 2023-11-15T04:33:08.012 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.008+0000 7f011f7fe700 1 --2- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 0x7f01200f6bd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f0108009830 tx=0x7f010800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:08.012 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.008+0000 7f0117fff700 1 -- 172.21.15.190:0/400713013 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0108009320 con 0x7f0120060220 2023-11-15T04:33:08.013 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.008+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01200f51f0 con 0x7f0120060220 2023-11-15T04:33:08.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.008+0000 7f0117fff700 1 -- 172.21.15.190:0/400713013 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01080136c0 con 0x7f0120060220 2023-11-15T04:33:08.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.008+0000 7f0117fff700 1 -- 172.21.15.190:0/400713013 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f010801ee90 con 0x7f0120060220 2023-11-15T04:33:08.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.008+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01200f5680 con 0x7f0120060220 2023-11-15T04:33:08.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.009+0000 7f0117fff700 1 -- 172.21.15.190:0/400713013 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f010801e420 con 0x7f0120060220 2023-11-15T04:33:08.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.009+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0120059100 con 0x7f0120060220 2023-11-15T04:33:08.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.009+0000 7f0117fff700 1 --2- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f010c02ff10 0x7f010c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:08.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.009+0000 7f0117fff700 1 -- 172.21.15.190:0/400713013 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0108040f40 con 0x7f0120060220 2023-11-15T04:33:08.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.009+0000 7f011effd700 1 --2- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f010c02ff10 0x7f010c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:08.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.009+0000 7f011effd700 1 --2- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f010c02ff10 0x7f010c0323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f01100096a0 tx=0x7f0110006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:08.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.014+0000 7f0117fff700 1 -- 172.21.15.190:0/400713013 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f010806d2b0 con 0x7f0120060220 2023-11-15T04:33:08.220 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.220+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01200fbf50 con 0x7f0120060220 2023-11-15T04:33:08.221 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.220+0000 7f0117fff700 1 -- 172.21.15.190:0/400713013 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01080095f0 con 0x7f0120060220 2023-11-15T04:33:08.221 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:08.222 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:08.224 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f010c02ff10 msgr2=0x7f010c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:08.224 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 --2- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f010c02ff10 0x7f010c0323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f01100096a0 tx=0x7f0110006b40).stop 2023-11-15T04:33:08.224 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 msgr2=0x7f01200f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:08.224 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 --2- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 0x7f01200f6bd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f0108009830 tx=0x7f010800b040).stop 2023-11-15T04:33:08.225 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 shutdown_connections 2023-11-15T04:33:08.225 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 --2- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f010c02ff10 0x7f010c0323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:08.225 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 --2- 172.21.15.190:0/400713013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0120060220 0x7f01200f6bd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:08.226 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 >> 172.21.15.190:0/400713013 conn(0x7f01200f0b70 msgr2=0x7f01200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:08.226 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 shutdown_connections 2023-11-15T04:33:08.226 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:08.222+0000 7f01261c5700 1 -- 172.21.15.190:0/400713013 wait complete. 2023-11-15T04:33:08.226 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:09.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:08.221668+0000 mon.smithi047 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.190:0/400713013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:09.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:09.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:10.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:09.093921+0000 mgr.smithi047.zgysuz (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:10.766 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.764+0000 7fc08a9aa700 1 -- 172.21.15.190:0/4285274656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 msgr2=0x7fc0840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:10.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.764+0000 7fc08a9aa700 1 --2- 172.21.15.190:0/4285274656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 0x7fc0840f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fc074009fb0 tx=0x7fc074009b30).stop 2023-11-15T04:33:10.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.764+0000 7fc08a9aa700 1 -- 172.21.15.190:0/4285274656 shutdown_connections 2023-11-15T04:33:10.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.764+0000 7fc08a9aa700 1 --2- 172.21.15.190:0/4285274656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 0x7fc0840f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:10.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.764+0000 7fc08a9aa700 1 -- 172.21.15.190:0/4285274656 >> 172.21.15.190:0/4285274656 conn(0x7fc0840f0b70 msgr2=0x7fc0840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:10.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.764+0000 7fc08a9aa700 1 -- 172.21.15.190:0/4285274656 shutdown_connections 2023-11-15T04:33:10.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.764+0000 7fc08a9aa700 1 -- 172.21.15.190:0/4285274656 wait complete. 2023-11-15T04:33:10.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.764+0000 7fc08a9aa700 1 Processor -- start 2023-11-15T04:33:10.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.765+0000 7fc08a9aa700 1 -- start start 2023-11-15T04:33:10.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.765+0000 7fc08a9aa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 0x7fc0840fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:10.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.765+0000 7fc08a9aa700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0840fb820 con 0x7fc0840f5410 2023-11-15T04:33:10.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.765+0000 7fc083fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 0x7fc0840fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:10.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.765+0000 7fc083fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 0x7fc0840fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34894/0 (socket says 172.21.15.190:34894) 2023-11-15T04:33:10.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.765+0000 7fc083fff700 1 -- 172.21.15.190:0/3976823879 learned_addr learned my addr 172.21.15.190:0/3976823879 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:10.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.765+0000 7fc083fff700 1 -- 172.21.15.190:0/3976823879 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc074005040 con 0x7fc0840f5410 2023-11-15T04:33:10.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.765+0000 7fc083fff700 1 --2- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 0x7fc0840fd3a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc074013510 tx=0x7fc07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:10.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.766+0000 7fc080ff9700 1 -- 172.21.15.190:0/3976823879 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc074009320 con 0x7fc0840f5410 2023-11-15T04:33:10.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.766+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0840fb9c0 con 0x7fc0840f5410 2023-11-15T04:33:10.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.766+0000 7fc080ff9700 1 -- 172.21.15.190:0/3976823879 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc074013f00 con 0x7fc0840f5410 2023-11-15T04:33:10.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.766+0000 7fc080ff9700 1 -- 172.21.15.190:0/3976823879 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc074014d90 con 0x7fc0840f5410 2023-11-15T04:33:10.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.766+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0840fbe50 con 0x7fc0840f5410 2023-11-15T04:33:10.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.766+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc084059100 con 0x7fc0840f5410 2023-11-15T04:33:10.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.767+0000 7fc080ff9700 1 -- 172.21.15.190:0/3976823879 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0740095f0 con 0x7fc0840f5410 2023-11-15T04:33:10.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.767+0000 7fc080ff9700 1 --2- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc06c02ff60 0x7fc06c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:10.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.767+0000 7fc080ff9700 1 -- 172.21.15.190:0/3976823879 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc074041000 con 0x7fc0840f5410 2023-11-15T04:33:10.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.767+0000 7fc0837fe700 1 --2- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc06c02ff60 0x7fc06c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:10.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.768+0000 7fc0837fe700 1 --2- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc06c02ff60 0x7fc06c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc0780096a0 tx=0x7fc078006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:10.773 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.772+0000 7fc080ff9700 1 -- 172.21.15.190:0/3976823879 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc074014420 con 0x7fc0840f5410 2023-11-15T04:33:10.979 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.978+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0840fae60 con 0x7fc0840f5410 2023-11-15T04:33:10.980 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.979+0000 7fc080ff9700 1 -- 172.21.15.190:0/3976823879 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc074010230 con 0x7fc0840f5410 2023-11-15T04:33:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:10.980 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:10.981 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc06c02ff60 msgr2=0x7fc06c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:10.982 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 --2- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc06c02ff60 0x7fc06c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc0780096a0 tx=0x7fc078006b40).stop 2023-11-15T04:33:10.982 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 msgr2=0x7fc0840fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:10.982 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 --2- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 0x7fc0840fd3a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc074013510 tx=0x7fc07400b040).stop 2023-11-15T04:33:10.982 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 shutdown_connections 2023-11-15T04:33:10.982 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 --2- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc06c02ff60 0x7fc06c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:10.983 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 --2- 172.21.15.190:0/3976823879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0840f5410 0x7fc0840fd3a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:10.983 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 >> 172.21.15.190:0/3976823879 conn(0x7fc0840f0b70 msgr2=0x7fc08405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:10.983 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 shutdown_connections 2023-11-15T04:33:10.983 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:10.980+0000 7fc08a9aa700 1 -- 172.21.15.190:0/3976823879 wait complete. 2023-11-15T04:33:10.984 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:11.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:10.980422+0000 mon.smithi047 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.190:0/3976823879' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:12.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:12.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:12.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:12 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:11.094274+0000 mgr.smithi047.zgysuz (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:13.502 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.501+0000 7f61ede70700 1 -- 172.21.15.190:0/2261011998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 msgr2=0x7f61e80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:13.503 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.501+0000 7f61ede70700 1 --2- 172.21.15.190:0/2261011998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 0x7f61e80f90f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f61d0009fb0 tx=0x7f61d0009b30).stop 2023-11-15T04:33:13.503 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.501+0000 7f61ede70700 1 -- 172.21.15.190:0/2261011998 shutdown_connections 2023-11-15T04:33:13.503 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.501+0000 7f61ede70700 1 --2- 172.21.15.190:0/2261011998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 0x7f61e80f90f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:13.503 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.501+0000 7f61ede70700 1 -- 172.21.15.190:0/2261011998 >> 172.21.15.190:0/2261011998 conn(0x7f61e80f0b70 msgr2=0x7f61e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:13.503 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.501+0000 7f61ede70700 1 -- 172.21.15.190:0/2261011998 shutdown_connections 2023-11-15T04:33:13.503 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.501+0000 7f61ede70700 1 -- 172.21.15.190:0/2261011998 wait complete. 2023-11-15T04:33:13.504 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.501+0000 7f61ede70700 1 Processor -- start 2023-11-15T04:33:13.504 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.501+0000 7f61ede70700 1 -- start start 2023-11-15T04:33:13.504 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.502+0000 7f61ede70700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 0x7f61e81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:13.504 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.502+0000 7f61ede70700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e8106bd0 con 0x7f61e8060280 2023-11-15T04:33:13.506 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.502+0000 7f61e77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 0x7f61e81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:13.506 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.502+0000 7f61e77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 0x7f61e81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34912/0 (socket says 172.21.15.190:34912) 2023-11-15T04:33:13.506 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.502+0000 7f61e77fe700 1 -- 172.21.15.190:0/2578823819 learned_addr learned my addr 172.21.15.190:0/2578823819 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:13.506 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.502+0000 7f61e77fe700 1 -- 172.21.15.190:0/2578823819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d0005040 con 0x7f61e8060280 2023-11-15T04:33:13.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.502+0000 7f61e77fe700 1 --2- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 0x7f61e81066e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f61d0013510 tx=0x7f61d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:13.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.502+0000 7f61dffff700 1 -- 172.21.15.190:0/2578823819 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d0009320 con 0x7f61e8060280 2023-11-15T04:33:13.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.502+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e8106dd0 con 0x7f61e8060280 2023-11-15T04:33:13.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.503+0000 7f61dffff700 1 -- 172.21.15.190:0/2578823819 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61d0013f00 con 0x7f61e8060280 2023-11-15T04:33:13.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.503+0000 7f61dffff700 1 -- 172.21.15.190:0/2578823819 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d0014dc0 con 0x7f61e8060280 2023-11-15T04:33:13.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.503+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e8107260 con 0x7f61e8060280 2023-11-15T04:33:13.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.503+0000 7f61dffff700 1 -- 172.21.15.190:0/2578823819 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61d00095f0 con 0x7f61e8060280 2023-11-15T04:33:13.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.503+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e8045ad0 con 0x7f61e8060280 2023-11-15T04:33:13.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.503+0000 7f61dffff700 1 --2- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61d402ff60 0x7f61d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:13.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.504+0000 7f61dffff700 1 -- 172.21.15.190:0/2578823819 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61d00410c0 con 0x7f61e8060280 2023-11-15T04:33:13.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.504+0000 7f61e6ffd700 1 --2- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61d402ff60 0x7f61d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:13.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.504+0000 7f61e6ffd700 1 --2- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61d402ff60 0x7f61d4032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f61d80096a0 tx=0x7f61d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:13.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.509+0000 7f61dffff700 1 -- 172.21.15.190:0/2578823819 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61d0014420 con 0x7f61e8060280 2023-11-15T04:33:13.716 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.715+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61e81080a0 con 0x7f61e8060280 2023-11-15T04:33:13.717 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.715+0000 7f61dffff700 1 -- 172.21.15.190:0/2578823819 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f61d0010230 con 0x7f61e8060280 2023-11-15T04:33:13.717 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:13.717 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:13.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61d402ff60 msgr2=0x7f61d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:13.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 --2- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61d402ff60 0x7f61d4032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f61d80096a0 tx=0x7f61d8006b40).stop 2023-11-15T04:33:13.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 msgr2=0x7f61e81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:13.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 --2- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 0x7f61e81066e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f61d0013510 tx=0x7f61d000b040).stop 2023-11-15T04:33:13.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 shutdown_connections 2023-11-15T04:33:13.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 --2- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61d402ff60 0x7f61d4032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:13.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 --2- 172.21.15.190:0/2578823819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e8060280 0x7f61e81066e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:13.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 >> 172.21.15.190:0/2578823819 conn(0x7f61e80f0b70 msgr2=0x7f61e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:13.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 shutdown_connections 2023-11-15T04:33:13.720 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:13.717+0000 7f61ede70700 1 -- 172.21.15.190:0/2578823819 wait complete. 2023-11-15T04:33:13.720 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:13.094668+0000 mgr.smithi047.zgysuz (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:13.717123+0000 mon.smithi047 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.190:0/2578823819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:15.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:15.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:16.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.277+0000 7fec18b21700 1 -- 172.21.15.190:0/3412915787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 msgr2=0x7fec140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:16.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.277+0000 7fec18b21700 1 --2- 172.21.15.190:0/3412915787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 0x7fec140f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fec04009fb0 tx=0x7fec04009b30).stop 2023-11-15T04:33:16.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.277+0000 7fec18b21700 1 -- 172.21.15.190:0/3412915787 shutdown_connections 2023-11-15T04:33:16.279 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.277+0000 7fec18b21700 1 --2- 172.21.15.190:0/3412915787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 0x7fec140f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:16.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.277+0000 7fec18b21700 1 -- 172.21.15.190:0/3412915787 >> 172.21.15.190:0/3412915787 conn(0x7fec140f1c70 msgr2=0x7fec140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:16.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.277+0000 7fec18b21700 1 -- 172.21.15.190:0/3412915787 shutdown_connections 2023-11-15T04:33:16.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.277+0000 7fec18b21700 1 -- 172.21.15.190:0/3412915787 wait complete. 2023-11-15T04:33:16.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.278+0000 7fec18b21700 1 Processor -- start 2023-11-15T04:33:16.280 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.278+0000 7fec18b21700 1 -- start start 2023-11-15T04:33:16.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.278+0000 7fec18b21700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 0x7fec14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:16.281 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.278+0000 7fec18b21700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec14106c20 con 0x7fec140f6510 2023-11-15T04:33:16.282 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.278+0000 7fec1259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 0x7fec14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:16.282 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.278+0000 7fec1259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 0x7fec14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34938/0 (socket says 172.21.15.190:34938) 2023-11-15T04:33:16.283 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.278+0000 7fec1259c700 1 -- 172.21.15.190:0/1178951022 learned_addr learned my addr 172.21.15.190:0/1178951022 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:16.283 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.279+0000 7fec1259c700 1 -- 172.21.15.190:0/1178951022 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec04005040 con 0x7fec140f6510 2023-11-15T04:33:16.283 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.279+0000 7fec1259c700 1 --2- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 0x7fec14106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fec04014000 tx=0x7fec0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:16.283 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.279+0000 7fec0affd700 1 -- 172.21.15.190:0/1178951022 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec040095f0 con 0x7fec140f6510 2023-11-15T04:33:16.284 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.279+0000 7fec0affd700 1 -- 172.21.15.190:0/1178951022 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec04007e70 con 0x7fec140f6510 2023-11-15T04:33:16.284 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.279+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec14106e20 con 0x7fec140f6510 2023-11-15T04:33:16.284 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.279+0000 7fec0affd700 1 -- 172.21.15.190:0/1178951022 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec04014ea0 con 0x7fec140f6510 2023-11-15T04:33:16.284 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.279+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec141072b0 con 0x7fec140f6510 2023-11-15T04:33:16.285 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.280+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec14045ad0 con 0x7fec140f6510 2023-11-15T04:33:16.285 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.280+0000 7fec0affd700 1 -- 172.21.15.190:0/1178951022 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec04014690 con 0x7fec140f6510 2023-11-15T04:33:16.285 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.280+0000 7fec0affd700 1 --2- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fec0002ff60 0x7fec00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:16.285 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.280+0000 7fec0affd700 1 -- 172.21.15.190:0/1178951022 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec04040930 con 0x7fec140f6510 2023-11-15T04:33:16.285 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.280+0000 7fec11d9b700 1 --2- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fec0002ff60 0x7fec00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:16.286 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.281+0000 7fec11d9b700 1 --2- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fec0002ff60 0x7fec00032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7febfc0096a0 tx=0x7febfc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:16.286 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.285+0000 7fec0affd700 1 -- 172.21.15.190:0/1178951022 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec0400f300 con 0x7fec140f6510 2023-11-15T04:33:16.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.492+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec140fbf60 con 0x7fec140f6510 2023-11-15T04:33:16.493 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.492+0000 7fec0affd700 1 -- 172.21.15.190:0/1178951022 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec0400f300 con 0x7fec140f6510 2023-11-15T04:33:16.494 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:16.494 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:16.495 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fec0002ff60 msgr2=0x7fec00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:16.495 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 --2- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fec0002ff60 0x7fec00032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7febfc0096a0 tx=0x7febfc006b40).stop 2023-11-15T04:33:16.495 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 msgr2=0x7fec14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:16.495 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 --2- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 0x7fec14106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fec04014000 tx=0x7fec0400b040).stop 2023-11-15T04:33:16.496 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 shutdown_connections 2023-11-15T04:33:16.496 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 --2- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fec0002ff60 0x7fec00032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:16.496 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 --2- 172.21.15.190:0/1178951022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec140f6510 0x7fec14106730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:16.496 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 >> 172.21.15.190:0/1178951022 conn(0x7fec140f1c70 msgr2=0x7fec140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:16.496 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 shutdown_connections 2023-11-15T04:33:16.497 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:16.494+0000 7fec18b21700 1 -- 172.21.15.190:0/1178951022 wait complete. 2023-11-15T04:33:16.497 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:15.095101+0000 mgr.smithi047.zgysuz (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:17.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:16.493790+0000 mon.smithi047 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.190:0/1178951022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:17.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:17.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:18.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:17.095486+0000 mgr.smithi047.zgysuz (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:18.997 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.995+0000 7f0053dc6700 1 -- 172.21.15.190:0/956749833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 msgr2=0x7f004c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:18.998 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.995+0000 7f0053dc6700 1 --2- 172.21.15.190:0/956749833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 0x7f004c0f4be0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0048009fb0 tx=0x7f0048009b30).stop 2023-11-15T04:33:18.998 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.995+0000 7f0053dc6700 1 -- 172.21.15.190:0/956749833 shutdown_connections 2023-11-15T04:33:18.998 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.995+0000 7f0053dc6700 1 --2- 172.21.15.190:0/956749833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 0x7f004c0f4be0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:18.998 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.995+0000 7f0053dc6700 1 -- 172.21.15.190:0/956749833 >> 172.21.15.190:0/956749833 conn(0x7f004c0f02e0 msgr2=0x7f004c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:18.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.995+0000 7f0053dc6700 1 -- 172.21.15.190:0/956749833 shutdown_connections 2023-11-15T04:33:18.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.995+0000 7f0053dc6700 1 -- 172.21.15.190:0/956749833 wait complete. 2023-11-15T04:33:18.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.996+0000 7f0053dc6700 1 Processor -- start 2023-11-15T04:33:18.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.996+0000 7f0053dc6700 1 -- start start 2023-11-15T04:33:19.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.996+0000 7f0053dc6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 0x7f004c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:19.002 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.996+0000 7f0053dc6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004c106bf0 con 0x7f004c0f47c0 2023-11-15T04:33:19.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.996+0000 7f0051b62700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 0x7f004c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:19.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.996+0000 7f0051b62700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 0x7f004c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34956/0 (socket says 172.21.15.190:34956) 2023-11-15T04:33:19.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.996+0000 7f0051b62700 1 -- 172.21.15.190:0/2674765451 learned_addr learned my addr 172.21.15.190:0/2674765451 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:19.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.996+0000 7f0051b62700 1 -- 172.21.15.190:0/2674765451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0048005040 con 0x7f004c0f47c0 2023-11-15T04:33:19.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.996+0000 7f0051b62700 1 --2- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 0x7f004c106700 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0048013510 tx=0x7f004800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:19.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.997+0000 7f00427fc700 1 -- 172.21.15.190:0/2674765451 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0048009320 con 0x7f004c0f47c0 2023-11-15T04:33:19.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.997+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f004c106df0 con 0x7f004c0f47c0 2023-11-15T04:33:19.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.997+0000 7f00427fc700 1 -- 172.21.15.190:0/2674765451 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0048013f00 con 0x7f004c0f47c0 2023-11-15T04:33:19.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.997+0000 7f00427fc700 1 -- 172.21.15.190:0/2674765451 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0048014d90 con 0x7f004c0f47c0 2023-11-15T04:33:19.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.997+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f004c107280 con 0x7f004c0f47c0 2023-11-15T04:33:19.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.998+0000 7f00427fc700 1 -- 172.21.15.190:0/2674765451 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00480095f0 con 0x7f004c0f47c0 2023-11-15T04:33:19.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.998+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004c045ad0 con 0x7f004c0f47c0 2023-11-15T04:33:19.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.998+0000 7f00427fc700 1 --2- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f003802ff60 0x7f0038032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:19.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.998+0000 7f00427fc700 1 -- 172.21.15.190:0/2674765451 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0048040fc0 con 0x7f004c0f47c0 2023-11-15T04:33:19.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.998+0000 7f0051361700 1 --2- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f003802ff60 0x7f0038032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:19.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:18.998+0000 7f0051361700 1 --2- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f003802ff60 0x7f0038032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f003c0096a0 tx=0x7f003c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:19.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.003+0000 7f00427fc700 1 -- 172.21.15.190:0/2674765451 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0048014420 con 0x7f004c0f47c0 2023-11-15T04:33:19.209 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.208+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f004c1080d0 con 0x7f004c0f47c0 2023-11-15T04:33:19.210 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.209+0000 7f00427fc700 1 -- 172.21.15.190:0/2674765451 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0048010230 con 0x7f004c0f47c0 2023-11-15T04:33:19.210 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:19.210 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:19.212 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f003802ff60 msgr2=0x7f0038032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:19.212 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 --2- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f003802ff60 0x7f0038032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f003c0096a0 tx=0x7f003c006b40).stop 2023-11-15T04:33:19.212 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 msgr2=0x7f004c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:19.212 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 --2- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 0x7f004c106700 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0048013510 tx=0x7f004800b040).stop 2023-11-15T04:33:19.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 shutdown_connections 2023-11-15T04:33:19.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 --2- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f003802ff60 0x7f0038032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:19.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 --2- 172.21.15.190:0/2674765451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f004c0f47c0 0x7f004c106700 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:19.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 >> 172.21.15.190:0/2674765451 conn(0x7f004c0f02e0 msgr2=0x7f004c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:19.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 shutdown_connections 2023-11-15T04:33:19.214 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:19.211+0000 7f0053dc6700 1 -- 172.21.15.190:0/2674765451 wait complete. 2023-11-15T04:33:19.214 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:20.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:20 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:19.095830+0000 mgr.smithi047.zgysuz (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:20.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:20 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:19.210374+0000 mon.smithi047 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.190:0/2674765451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:20.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:20.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:21.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 -- 172.21.15.190:0/2227613259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 msgr2=0x7f4c900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:21.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 --2- 172.21.15.190:0/2227613259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 0x7f4c900f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f4c80009fb0 tx=0x7f4c80009b30).stop 2023-11-15T04:33:21.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 -- 172.21.15.190:0/2227613259 shutdown_connections 2023-11-15T04:33:21.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 --2- 172.21.15.190:0/2227613259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 0x7f4c900f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:21.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 -- 172.21.15.190:0/2227613259 >> 172.21.15.190:0/2227613259 conn(0x7f4c900f0b70 msgr2=0x7f4c900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:21.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 -- 172.21.15.190:0/2227613259 shutdown_connections 2023-11-15T04:33:21.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 -- 172.21.15.190:0/2227613259 wait complete. 2023-11-15T04:33:21.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 Processor -- start 2023-11-15T04:33:21.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 -- start start 2023-11-15T04:33:21.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 0x7f4c9010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:21.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.768+0000 7f4c948c2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c90107240 con 0x7f4c900f5410 2023-11-15T04:33:21.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c8e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 0x7f4c9010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:21.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c8e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 0x7f4c9010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:39822/0 (socket says 172.21.15.190:39822) 2023-11-15T04:33:21.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c8e59c700 1 -- 172.21.15.190:0/2202903845 learned_addr learned my addr 172.21.15.190:0/2202903845 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:21.773 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c8e59c700 1 -- 172.21.15.190:0/2202903845 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c80005040 con 0x7f4c900f5410 2023-11-15T04:33:21.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c8e59c700 1 --2- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 0x7f4c9010b850 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f4c80014000 tx=0x7f4c8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:21.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c86ffd700 1 -- 172.21.15.190:0/2202903845 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c800095f0 con 0x7f4c900f5410 2023-11-15T04:33:21.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c90060220 con 0x7f4c900f5410 2023-11-15T04:33:21.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c86ffd700 1 -- 172.21.15.190:0/2202903845 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c80007e70 con 0x7f4c900f5410 2023-11-15T04:33:21.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9010bd40 con 0x7f4c900f5410 2023-11-15T04:33:21.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.769+0000 7f4c86ffd700 1 -- 172.21.15.190:0/2202903845 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c80014ea0 con 0x7f4c900f5410 2023-11-15T04:33:21.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.770+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c90045ad0 con 0x7f4c900f5410 2023-11-15T04:33:21.776 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.770+0000 7f4c86ffd700 1 -- 172.21.15.190:0/2202903845 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c80025070 con 0x7f4c900f5410 2023-11-15T04:33:21.776 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.770+0000 7f4c86ffd700 1 --2- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4c7c02ff60 0x7f4c7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:21.777 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.770+0000 7f4c86ffd700 1 -- 172.21.15.190:0/2202903845 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c800409d0 con 0x7f4c900f5410 2023-11-15T04:33:21.777 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.770+0000 7f4c87fff700 1 --2- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4c7c02ff60 0x7f4c7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:21.777 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.771+0000 7f4c87fff700 1 --2- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4c7c02ff60 0x7f4c7c032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4c780096a0 tx=0x7f4c78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:21.778 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.776+0000 7f4c86ffd700 1 -- 172.21.15.190:0/2202903845 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c800079c0 con 0x7f4c900f5410 2023-11-15T04:33:21.983 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.981+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c900fae60 con 0x7f4c900f5410 2023-11-15T04:33:21.983 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c86ffd700 1 -- 172.21.15.190:0/2202903845 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c8000f300 con 0x7f4c900f5410 2023-11-15T04:33:21.983 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:21.984 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:21.985 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4c7c02ff60 msgr2=0x7f4c7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:21.985 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 --2- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4c7c02ff60 0x7f4c7c032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4c780096a0 tx=0x7f4c78006b40).stop 2023-11-15T04:33:21.985 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 msgr2=0x7f4c9010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:21.985 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 --2- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 0x7f4c9010b850 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f4c80014000 tx=0x7f4c8000b040).stop 2023-11-15T04:33:21.986 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 shutdown_connections 2023-11-15T04:33:21.986 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 --2- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4c7c02ff60 0x7f4c7c032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:21.986 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 --2- 172.21.15.190:0/2202903845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c900f5410 0x7f4c9010b850 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:21.986 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 >> 172.21.15.190:0/2202903845 conn(0x7f4c900f0b70 msgr2=0x7f4c900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:21.986 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 shutdown_connections 2023-11-15T04:33:21.986 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:21.983+0000 7f4c948c2700 1 -- 172.21.15.190:0/2202903845 wait complete. 2023-11-15T04:33:21.987 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:21.096216+0000 mgr.smithi047.zgysuz (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:21.983358+0000 mon.smithi047 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.190:0/2202903845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:23.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:23.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:24.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.476+0000 7fa3b0997700 1 -- 172.21.15.190:0/3376729940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 msgr2=0x7fa3ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:24.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.476+0000 7fa3b0997700 1 --2- 172.21.15.190:0/3376729940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 0x7fa3ac0f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fa3a0009fb0 tx=0x7fa3a0009b30).stop 2023-11-15T04:33:24.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.476+0000 7fa3b0997700 1 -- 172.21.15.190:0/3376729940 shutdown_connections 2023-11-15T04:33:24.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.476+0000 7fa3b0997700 1 --2- 172.21.15.190:0/3376729940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 0x7fa3ac0f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:24.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.476+0000 7fa3b0997700 1 -- 172.21.15.190:0/3376729940 >> 172.21.15.190:0/3376729940 conn(0x7fa3ac0f0b70 msgr2=0x7fa3ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:24.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.476+0000 7fa3b0997700 1 -- 172.21.15.190:0/3376729940 shutdown_connections 2023-11-15T04:33:24.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.476+0000 7fa3b0997700 1 -- 172.21.15.190:0/3376729940 wait complete. 2023-11-15T04:33:24.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.477+0000 7fa3b0997700 1 Processor -- start 2023-11-15T04:33:24.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.477+0000 7fa3b0997700 1 -- start start 2023-11-15T04:33:24.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.477+0000 7fa3b0997700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 0x7fa3ac109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:24.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.477+0000 7fa3b0997700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac060220 con 0x7fa3ac0f5410 2023-11-15T04:33:24.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.477+0000 7fa3aa59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 0x7fa3ac109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:24.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.477+0000 7fa3aa59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 0x7fa3ac109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:39850/0 (socket says 172.21.15.190:39850) 2023-11-15T04:33:24.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.477+0000 7fa3aa59c700 1 -- 172.21.15.190:0/4177361182 learned_addr learned my addr 172.21.15.190:0/4177361182 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:24.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.478+0000 7fa3aa59c700 1 -- 172.21.15.190:0/4177361182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a0005040 con 0x7fa3ac0f5410 2023-11-15T04:33:24.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.478+0000 7fa3aa59c700 1 --2- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 0x7fa3ac109420 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fa3a0013510 tx=0x7fa3a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:24.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.478+0000 7fa39affd700 1 -- 172.21.15.190:0/4177361182 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a0009320 con 0x7fa3ac0f5410 2023-11-15T04:33:24.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.478+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac060420 con 0x7fa3ac0f5410 2023-11-15T04:33:24.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.478+0000 7fa39affd700 1 -- 172.21.15.190:0/4177361182 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3a0013f00 con 0x7fa3ac0f5410 2023-11-15T04:33:24.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.478+0000 7fa39affd700 1 -- 172.21.15.190:0/4177361182 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a0014d90 con 0x7fa3ac0f5410 2023-11-15T04:33:24.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.478+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac109ad0 con 0x7fa3ac0f5410 2023-11-15T04:33:24.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.479+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac059100 con 0x7fa3ac0f5410 2023-11-15T04:33:24.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.479+0000 7fa39affd700 1 -- 172.21.15.190:0/4177361182 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3a0014420 con 0x7fa3ac0f5410 2023-11-15T04:33:24.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.479+0000 7fa39affd700 1 --2- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39402ff60 0x7fa394032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:24.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.479+0000 7fa39affd700 1 -- 172.21.15.190:0/4177361182 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3a0040fb0 con 0x7fa3ac0f5410 2023-11-15T04:33:24.485 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.479+0000 7fa3a9d9b700 1 --2- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39402ff60 0x7fa394032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:24.485 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.480+0000 7fa3a9d9b700 1 --2- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39402ff60 0x7fa394032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa39c0096a0 tx=0x7fa39c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:24.485 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.484+0000 7fa39affd700 1 -- 172.21.15.190:0/4177361182 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3a00095f0 con 0x7fa3ac0f5410 2023-11-15T04:33:24.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:23.096561+0000 mgr.smithi047.zgysuz (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:24.688 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.688+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3ac045ad0 con 0x7fa3ac0f5410 2023-11-15T04:33:24.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.689+0000 7fa39affd700 1 -- 172.21.15.190:0/4177361182 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3a001db70 con 0x7fa3ac0f5410 2023-11-15T04:33:24.689 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:24.689 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:24.690 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39402ff60 msgr2=0x7fa394032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:24.690 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 --2- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39402ff60 0x7fa394032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa39c0096a0 tx=0x7fa39c006b40).stop 2023-11-15T04:33:24.691 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 msgr2=0x7fa3ac109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:24.691 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 --2- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 0x7fa3ac109420 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fa3a0013510 tx=0x7fa3a000b040).stop 2023-11-15T04:33:24.691 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 shutdown_connections 2023-11-15T04:33:24.691 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 --2- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39402ff60 0x7fa394032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:24.691 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 --2- 172.21.15.190:0/4177361182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3ac0f5410 0x7fa3ac109420 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:24.692 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 >> 172.21.15.190:0/4177361182 conn(0x7fa3ac0f0b70 msgr2=0x7fa3ac0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:24.692 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 shutdown_connections 2023-11-15T04:33:24.692 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:24.690+0000 7fa3b0997700 1 -- 172.21.15.190:0/4177361182 wait complete. 2023-11-15T04:33:24.693 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:25.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:25 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:24.689256+0000 mon.smithi047 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.190:0/4177361182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:26.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:26.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:26.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:25.096904+0000 mgr.smithi047.zgysuz (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:27.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.151+0000 7fc5c227b700 1 -- 172.21.15.190:0/1341329596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 msgr2=0x7fc5bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:27.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.151+0000 7fc5c227b700 1 --2- 172.21.15.190:0/1341329596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 0x7fc5bc0f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fc5ac009fb0 tx=0x7fc5ac009b30).stop 2023-11-15T04:33:27.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.151+0000 7fc5c227b700 1 -- 172.21.15.190:0/1341329596 shutdown_connections 2023-11-15T04:33:27.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.151+0000 7fc5c227b700 1 --2- 172.21.15.190:0/1341329596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 0x7fc5bc0f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:27.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.151+0000 7fc5c227b700 1 -- 172.21.15.190:0/1341329596 >> 172.21.15.190:0/1341329596 conn(0x7fc5bc0f0b70 msgr2=0x7fc5bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:27.153 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.151+0000 7fc5c227b700 1 -- 172.21.15.190:0/1341329596 shutdown_connections 2023-11-15T04:33:27.153 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.151+0000 7fc5c227b700 1 -- 172.21.15.190:0/1341329596 wait complete. 2023-11-15T04:33:27.154 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.152+0000 7fc5c227b700 1 Processor -- start 2023-11-15T04:33:27.155 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.152+0000 7fc5c227b700 1 -- start start 2023-11-15T04:33:27.155 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5c227b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 0x7fc5bc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:27.155 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5c227b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc060240 con 0x7fc5bc0f5410 2023-11-15T04:33:27.155 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5bbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 0x7fc5bc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:27.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5bbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 0x7fc5bc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:39864/0 (socket says 172.21.15.190:39864) 2023-11-15T04:33:27.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5bbfff700 1 -- 172.21.15.190:0/2194045001 learned_addr learned my addr 172.21.15.190:0/2194045001 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:27.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5bbfff700 1 -- 172.21.15.190:0/2194045001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac005040 con 0x7fc5bc0f5410 2023-11-15T04:33:27.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5bbfff700 1 --2- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 0x7fc5bc109430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc5ac013510 tx=0x7fc5ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:27.156 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5b8ff9700 1 -- 172.21.15.190:0/2194045001 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ac009320 con 0x7fc5bc0f5410 2023-11-15T04:33:27.157 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc060440 con 0x7fc5bc0f5410 2023-11-15T04:33:27.157 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.153+0000 7fc5b8ff9700 1 -- 172.21.15.190:0/2194045001 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5ac013f00 con 0x7fc5bc0f5410 2023-11-15T04:33:27.157 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.154+0000 7fc5b8ff9700 1 -- 172.21.15.190:0/2194045001 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ac014d90 con 0x7fc5bc0f5410 2023-11-15T04:33:27.157 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.154+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc109ae0 con 0x7fc5bc0f5410 2023-11-15T04:33:27.157 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.154+0000 7fc5b8ff9700 1 -- 172.21.15.190:0/2194045001 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5ac0263a0 con 0x7fc5bc0f5410 2023-11-15T04:33:27.157 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.154+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc045ad0 con 0x7fc5bc0f5410 2023-11-15T04:33:27.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.155+0000 7fc5b8ff9700 1 --2- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc5a402ff60 0x7fc5a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:27.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.155+0000 7fc5b8ff9700 1 -- 172.21.15.190:0/2194045001 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5ac0412e0 con 0x7fc5bc0f5410 2023-11-15T04:33:27.158 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.155+0000 7fc5bb7fe700 1 --2- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc5a402ff60 0x7fc5a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:27.159 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.155+0000 7fc5bb7fe700 1 --2- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc5a402ff60 0x7fc5a4032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc5b00096a0 tx=0x7fc5b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:27.159 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.160+0000 7fc5b8ff9700 1 -- 172.21.15.190:0/2194045001 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5ac01cb10 con 0x7fc5bc0f5410 2023-11-15T04:33:27.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.365+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5bc0fae60 con 0x7fc5bc0f5410 2023-11-15T04:33:27.366 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.366+0000 7fc5b8ff9700 1 -- 172.21.15.190:0/2194045001 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc5ac01cb10 con 0x7fc5bc0f5410 2023-11-15T04:33:27.366 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:27.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.367+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc5a402ff60 msgr2=0x7fc5a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:27.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.367+0000 7fc5c227b700 1 --2- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc5a402ff60 0x7fc5a4032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc5b00096a0 tx=0x7fc5b0006b40).stop 2023-11-15T04:33:27.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.367+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 msgr2=0x7fc5bc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:27.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.367+0000 7fc5c227b700 1 --2- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 0x7fc5bc109430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc5ac013510 tx=0x7fc5ac00b040).stop 2023-11-15T04:33:27.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.368+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 shutdown_connections 2023-11-15T04:33:27.369 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.368+0000 7fc5c227b700 1 --2- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc5a402ff60 0x7fc5a4032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:27.369 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.368+0000 7fc5c227b700 1 --2- 172.21.15.190:0/2194045001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5bc0f5410 0x7fc5bc109430 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:27.369 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.368+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 >> 172.21.15.190:0/2194045001 conn(0x7fc5bc0f0b70 msgr2=0x7fc5bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:27.369 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.368+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 shutdown_connections 2023-11-15T04:33:27.369 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:27.368+0000 7fc5c227b700 1 -- 172.21.15.190:0/2194045001 wait complete. 2023-11-15T04:33:27.370 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:27.097241+0000 mgr.smithi047.zgysuz (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:27.366564+0000 mon.smithi047 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.190:0/2194045001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:28.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:28.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:29.849 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.849+0000 7f8b68bdb700 1 -- 172.21.15.190:0/3423435789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 msgr2=0x7f8b640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:29.849 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.849+0000 7f8b68bdb700 1 --2- 172.21.15.190:0/3423435789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 0x7f8b640f55c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f8b54009fb0 tx=0x7f8b54009b30).stop 2023-11-15T04:33:29.849 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.849+0000 7f8b68bdb700 1 -- 172.21.15.190:0/3423435789 shutdown_connections 2023-11-15T04:33:29.849 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.849+0000 7f8b68bdb700 1 --2- 172.21.15.190:0/3423435789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 0x7f8b640f55c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:29.850 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.849+0000 7f8b68bdb700 1 -- 172.21.15.190:0/3423435789 >> 172.21.15.190:0/3423435789 conn(0x7f8b640f09a0 msgr2=0x7f8b640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:29.850 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.849+0000 7f8b68bdb700 1 -- 172.21.15.190:0/3423435789 shutdown_connections 2023-11-15T04:33:29.850 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.849+0000 7f8b68bdb700 1 -- 172.21.15.190:0/3423435789 wait complete. 2023-11-15T04:33:29.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.849+0000 7f8b68bdb700 1 Processor -- start 2023-11-15T04:33:29.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.850+0000 7f8b68bdb700 1 -- start start 2023-11-15T04:33:29.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.850+0000 7f8b68bdb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 0x7f8b641086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:29.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.850+0000 7f8b68bdb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64108bb0 con 0x7f8b640f51a0 2023-11-15T04:33:29.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.850+0000 7f8b637fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 0x7f8b641086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:29.854 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.850+0000 7f8b637fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 0x7f8b641086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:39890/0 (socket says 172.21.15.190:39890) 2023-11-15T04:33:29.854 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.850+0000 7f8b637fe700 1 -- 172.21.15.190:0/2661170455 learned_addr learned my addr 172.21.15.190:0/2661170455 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:29.854 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.850+0000 7f8b637fe700 1 -- 172.21.15.190:0/2661170455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b54005040 con 0x7f8b640f51a0 2023-11-15T04:33:29.856 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.851+0000 7f8b637fe700 1 --2- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 0x7f8b641086c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f8b54009830 tx=0x7f8b5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:29.857 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.851+0000 7f8b4bfff700 1 -- 172.21.15.190:0/2661170455 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b54009320 con 0x7f8b640f51a0 2023-11-15T04:33:29.857 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.851+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b64108db0 con 0x7f8b640f51a0 2023-11-15T04:33:29.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.851+0000 7f8b4bfff700 1 -- 172.21.15.190:0/2661170455 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b540136c0 con 0x7f8b640f51a0 2023-11-15T04:33:29.859 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.851+0000 7f8b4bfff700 1 -- 172.21.15.190:0/2661170455 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b5401ee90 con 0x7f8b640f51a0 2023-11-15T04:33:29.859 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.851+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b64109240 con 0x7f8b640f51a0 2023-11-15T04:33:29.859 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.852+0000 7f8b4bfff700 1 -- 172.21.15.190:0/2661170455 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b54014440 con 0x7f8b640f51a0 2023-11-15T04:33:29.859 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.852+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b64045ad0 con 0x7f8b640f51a0 2023-11-15T04:33:29.860 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.852+0000 7f8b4bfff700 1 --2- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8b4c02ff10 0x7f8b4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:29.860 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.852+0000 7f8b4bfff700 1 -- 172.21.15.190:0/2661170455 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b540412a0 con 0x7f8b640f51a0 2023-11-15T04:33:29.860 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.852+0000 7f8b62ffd700 1 --2- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8b4c02ff10 0x7f8b4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:29.860 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.852+0000 7f8b62ffd700 1 --2- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8b4c02ff10 0x7f8b4c0323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f8b580096a0 tx=0x7f8b58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:29.861 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:29.857+0000 7f8b4bfff700 1 -- 172.21.15.190:0/2661170455 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b5401c910 con 0x7f8b640f51a0 2023-11-15T04:33:30.061 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.061+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b6410a0f0 con 0x7f8b640f51a0 2023-11-15T04:33:30.062 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.062+0000 7f8b4bfff700 1 -- 172.21.15.190:0/2661170455 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8b5401c320 con 0x7f8b640f51a0 2023-11-15T04:33:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:30.063 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:30.065 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8b4c02ff10 msgr2=0x7f8b4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:30.065 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 --2- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8b4c02ff10 0x7f8b4c0323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f8b580096a0 tx=0x7f8b58006b40).stop 2023-11-15T04:33:30.065 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 msgr2=0x7f8b641086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:30.065 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 --2- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 0x7f8b641086c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f8b54009830 tx=0x7f8b5400b040).stop 2023-11-15T04:33:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 shutdown_connections 2023-11-15T04:33:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 --2- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8b4c02ff10 0x7f8b4c0323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 --2- 172.21.15.190:0/2661170455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b640f51a0 0x7f8b641086c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 >> 172.21.15.190:0/2661170455 conn(0x7f8b640f09a0 msgr2=0x7f8b6405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 shutdown_connections 2023-11-15T04:33:30.067 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:30.064+0000 7f8b68bdb700 1 -- 172.21.15.190:0/2661170455 wait complete. 2023-11-15T04:33:30.067 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:30.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:29.097600+0000 mgr.smithi047.zgysuz (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:30.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:30.062766+0000 mon.smithi047 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.190:0/2661170455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:31.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:31.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:31.097983+0000 mgr.smithi047.zgysuz (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:32.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.586+0000 7f6e8b63a700 1 -- 172.21.15.190:0/2028267368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 msgr2=0x7f6e840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:32.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.586+0000 7f6e8b63a700 1 --2- 172.21.15.190:0/2028267368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 0x7f6e840f90f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f6e74009fb0 tx=0x7f6e74009b30).stop 2023-11-15T04:33:32.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.586+0000 7f6e8b63a700 1 -- 172.21.15.190:0/2028267368 shutdown_connections 2023-11-15T04:33:32.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.586+0000 7f6e8b63a700 1 --2- 172.21.15.190:0/2028267368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 0x7f6e840f90f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:32.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.586+0000 7f6e8b63a700 1 -- 172.21.15.190:0/2028267368 >> 172.21.15.190:0/2028267368 conn(0x7f6e840f0b70 msgr2=0x7f6e840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:32.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.586+0000 7f6e8b63a700 1 -- 172.21.15.190:0/2028267368 shutdown_connections 2023-11-15T04:33:32.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.586+0000 7f6e8b63a700 1 -- 172.21.15.190:0/2028267368 wait complete. 2023-11-15T04:33:32.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.587+0000 7f6e8b63a700 1 Processor -- start 2023-11-15T04:33:32.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.587+0000 7f6e8b63a700 1 -- start start 2023-11-15T04:33:32.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.587+0000 7f6e8b63a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 0x7f6e841088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:32.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.587+0000 7f6e8b63a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84108de0 con 0x7f6e84060280 2023-11-15T04:33:32.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.587+0000 7f6e893d6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 0x7f6e841088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:32.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.587+0000 7f6e893d6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 0x7f6e841088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:36166/0 (socket says 172.21.15.190:36166) 2023-11-15T04:33:32.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.587+0000 7f6e893d6700 1 -- 172.21.15.190:0/3926665402 learned_addr learned my addr 172.21.15.190:0/3926665402 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:32.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.587+0000 7f6e893d6700 1 -- 172.21.15.190:0/3926665402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74005040 con 0x7f6e84060280 2023-11-15T04:33:32.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.588+0000 7f6e893d6700 1 --2- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 0x7f6e841088f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f6e74013510 tx=0x7f6e7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:32.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.588+0000 7f6e79ffb700 1 -- 172.21.15.190:0/3926665402 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e74009320 con 0x7f6e84060280 2023-11-15T04:33:32.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.588+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e84108fe0 con 0x7f6e84060280 2023-11-15T04:33:32.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.588+0000 7f6e79ffb700 1 -- 172.21.15.190:0/3926665402 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e74013f00 con 0x7f6e84060280 2023-11-15T04:33:32.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.588+0000 7f6e79ffb700 1 -- 172.21.15.190:0/3926665402 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e74014dc0 con 0x7f6e84060280 2023-11-15T04:33:32.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.588+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e84109470 con 0x7f6e84060280 2023-11-15T04:33:32.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.589+0000 7f6e79ffb700 1 -- 172.21.15.190:0/3926665402 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e740095f0 con 0x7f6e84060280 2023-11-15T04:33:32.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.589+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e84045ad0 con 0x7f6e84060280 2023-11-15T04:33:32.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.589+0000 7f6e79ffb700 1 --2- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6e7002ff60 0x7f6e70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:32.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.589+0000 7f6e79ffb700 1 -- 172.21.15.190:0/3926665402 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e740410e0 con 0x7f6e84060280 2023-11-15T04:33:32.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.589+0000 7f6e88bd5700 1 --2- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6e7002ff60 0x7f6e70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:32.596 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.590+0000 7f6e88bd5700 1 --2- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6e7002ff60 0x7f6e70032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f6e800096a0 tx=0x7f6e80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:32.596 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.594+0000 7f6e79ffb700 1 -- 172.21.15.190:0/3926665402 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e74014420 con 0x7f6e84060280 2023-11-15T04:33:32.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.799+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e8410a2c0 con 0x7f6e84060280 2023-11-15T04:33:32.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.800+0000 7f6e79ffb700 1 -- 172.21.15.190:0/3926665402 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e7401c920 con 0x7f6e84060280 2023-11-15T04:33:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:32.801 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:32.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6e7002ff60 msgr2=0x7f6e70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:32.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 --2- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6e7002ff60 0x7f6e70032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f6e800096a0 tx=0x7f6e80006b40).stop 2023-11-15T04:33:32.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 msgr2=0x7f6e841088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:32.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 --2- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 0x7f6e841088f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f6e74013510 tx=0x7f6e7400b040).stop 2023-11-15T04:33:32.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 shutdown_connections 2023-11-15T04:33:32.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 --2- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6e7002ff60 0x7f6e70032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:32.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 --2- 172.21.15.190:0/3926665402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e84060280 0x7f6e841088f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:32.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 >> 172.21.15.190:0/3926665402 conn(0x7f6e840f0b70 msgr2=0x7f6e840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:32.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 shutdown_connections 2023-11-15T04:33:32.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:32.802+0000 7f6e8b63a700 1 -- 172.21.15.190:0/3926665402 wait complete. 2023-11-15T04:33:32.804 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:33.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:33 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:32.800734+0000 mon.smithi047 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.190:0/3926665402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:34.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:34.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:34.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:33.098312+0000 mgr.smithi047.zgysuz (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:35.370 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.368+0000 7f7f27076700 1 -- 172.21.15.190:0/408966822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 msgr2=0x7f7f200f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:35.370 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.368+0000 7f7f27076700 1 --2- 172.21.15.190:0/408966822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 0x7f7f200f6e30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f7f14009fb0 tx=0x7f7f14009b30).stop 2023-11-15T04:33:35.370 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.368+0000 7f7f27076700 1 -- 172.21.15.190:0/408966822 shutdown_connections 2023-11-15T04:33:35.370 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.368+0000 7f7f27076700 1 --2- 172.21.15.190:0/408966822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 0x7f7f200f6e30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:35.371 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.368+0000 7f7f27076700 1 -- 172.21.15.190:0/408966822 >> 172.21.15.190:0/408966822 conn(0x7f7f200f02e0 msgr2=0x7f7f200f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:35.371 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.368+0000 7f7f27076700 1 -- 172.21.15.190:0/408966822 shutdown_connections 2023-11-15T04:33:35.371 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.368+0000 7f7f27076700 1 -- 172.21.15.190:0/408966822 wait complete. 2023-11-15T04:33:35.371 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.369+0000 7f7f27076700 1 Processor -- start 2023-11-15T04:33:35.371 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.369+0000 7f7f27076700 1 -- start start 2023-11-15T04:33:35.372 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.369+0000 7f7f27076700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 0x7f7f201093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:35.372 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.369+0000 7f7f27076700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f200f5330 con 0x7f7f200f6a10 2023-11-15T04:33:35.372 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.369+0000 7f7f24e12700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 0x7f7f201093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:35.372 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.369+0000 7f7f24e12700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 0x7f7f201093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:36178/0 (socket says 172.21.15.190:36178) 2023-11-15T04:33:35.372 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.369+0000 7f7f24e12700 1 -- 172.21.15.190:0/1655187776 learned_addr learned my addr 172.21.15.190:0/1655187776 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:35.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.370+0000 7f7f24e12700 1 -- 172.21.15.190:0/1655187776 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14005040 con 0x7f7f200f6a10 2023-11-15T04:33:35.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.370+0000 7f7f24e12700 1 --2- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 0x7f7f201093f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7f14013510 tx=0x7f7f1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:35.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.370+0000 7f7f1d7fa700 1 -- 172.21.15.190:0/1655187776 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f14009320 con 0x7f7f200f6a10 2023-11-15T04:33:35.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.370+0000 7f7f1d7fa700 1 -- 172.21.15.190:0/1655187776 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f14013f00 con 0x7f7f200f6a10 2023-11-15T04:33:35.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.370+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f200f5530 con 0x7f7f200f6a10 2023-11-15T04:33:35.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.370+0000 7f7f1d7fa700 1 -- 172.21.15.190:0/1655187776 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f14014d90 con 0x7f7f200f6a10 2023-11-15T04:33:35.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.370+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f20109aa0 con 0x7f7f200f6a10 2023-11-15T04:33:35.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.371+0000 7f7f1d7fa700 1 -- 172.21.15.190:0/1655187776 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f14014420 con 0x7f7f200f6a10 2023-11-15T04:33:35.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.371+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f20059100 con 0x7f7f200f6a10 2023-11-15T04:33:35.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.371+0000 7f7f1d7fa700 1 --2- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7f1002ff60 0x7f7f10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:35.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.371+0000 7f7f1d7fa700 1 -- 172.21.15.190:0/1655187776 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f140410c0 con 0x7f7f200f6a10 2023-11-15T04:33:35.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.371+0000 7f7f1ffff700 1 --2- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7f1002ff60 0x7f7f10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:35.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.372+0000 7f7f1ffff700 1 --2- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7f1002ff60 0x7f7f10032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7f0c0096a0 tx=0x7f7f0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:35.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.376+0000 7f7f1d7fa700 1 -- 172.21.15.190:0/1655187776 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f140095f0 con 0x7f7f200f6a10 2023-11-15T04:33:35.582 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.582+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f20045ad0 con 0x7f7f200f6a10 2023-11-15T04:33:35.583 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.583+0000 7f7f1d7fa700 1 -- 172.21.15.190:0/1655187776 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f1401db70 con 0x7f7f200f6a10 2023-11-15T04:33:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:35.583 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:35.584 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.584+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7f1002ff60 msgr2=0x7f7f10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:35.584 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.584+0000 7f7f27076700 1 --2- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7f1002ff60 0x7f7f10032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7f0c0096a0 tx=0x7f7f0c006b40).stop 2023-11-15T04:33:35.585 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.584+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 msgr2=0x7f7f201093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:35.585 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.584+0000 7f7f27076700 1 --2- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 0x7f7f201093f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7f14013510 tx=0x7f7f1400b040).stop 2023-11-15T04:33:35.585 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.584+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 shutdown_connections 2023-11-15T04:33:35.585 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.584+0000 7f7f27076700 1 --2- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7f1002ff60 0x7f7f10032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:35.585 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.584+0000 7f7f27076700 1 --2- 172.21.15.190:0/1655187776 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f200f6a10 0x7f7f201093f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:35.586 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.584+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 >> 172.21.15.190:0/1655187776 conn(0x7f7f200f02e0 msgr2=0x7f7f200f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:35.586 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.585+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 shutdown_connections 2023-11-15T04:33:35.586 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:35.585+0000 7f7f27076700 1 -- 172.21.15.190:0/1655187776 wait complete. 2023-11-15T04:33:35.587 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:35.098637+0000 mgr.smithi047.zgysuz (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:35.583437+0000 mon.smithi047 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.190:0/1655187776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:36.967 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:36.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:38.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:37.099040+0000 mgr.smithi047.zgysuz (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:38.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.118+0000 7fd94171b700 1 -- 172.21.15.190:0/528203519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 msgr2=0x7fd93c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:38.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.118+0000 7fd94171b700 1 --2- 172.21.15.190:0/528203519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 0x7fd93c0f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fd930009fb0 tx=0x7fd930009b30).stop 2023-11-15T04:33:38.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.118+0000 7fd94171b700 1 -- 172.21.15.190:0/528203519 shutdown_connections 2023-11-15T04:33:38.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.118+0000 7fd94171b700 1 --2- 172.21.15.190:0/528203519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 0x7fd93c0f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:38.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.118+0000 7fd94171b700 1 -- 172.21.15.190:0/528203519 >> 172.21.15.190:0/528203519 conn(0x7fd93c0f0b70 msgr2=0x7fd93c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:38.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.118+0000 7fd94171b700 1 -- 172.21.15.190:0/528203519 shutdown_connections 2023-11-15T04:33:38.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.118+0000 7fd94171b700 1 -- 172.21.15.190:0/528203519 wait complete. 2023-11-15T04:33:38.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.119+0000 7fd94171b700 1 Processor -- start 2023-11-15T04:33:38.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.119+0000 7fd94171b700 1 -- start start 2023-11-15T04:33:38.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.119+0000 7fd94171b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 0x7fd93c0fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:38.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.119+0000 7fd94171b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0f9a00 con 0x7fd93c0f5410 2023-11-15T04:33:38.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.119+0000 7fd93affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 0x7fd93c0fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:38.122 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.119+0000 7fd93affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 0x7fd93c0fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:36188/0 (socket says 172.21.15.190:36188) 2023-11-15T04:33:38.122 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.119+0000 7fd93affd700 1 -- 172.21.15.190:0/1293842018 learned_addr learned my addr 172.21.15.190:0/1293842018 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:38.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.119+0000 7fd93affd700 1 -- 172.21.15.190:0/1293842018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd930005040 con 0x7fd93c0f5410 2023-11-15T04:33:38.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.119+0000 7fd93affd700 1 --2- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 0x7fd93c0fb580 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fd930013510 tx=0x7fd93000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:38.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.120+0000 7fd9237fe700 1 -- 172.21.15.190:0/1293842018 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd930009320 con 0x7fd93c0f5410 2023-11-15T04:33:38.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.120+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c0f9ba0 con 0x7fd93c0f5410 2023-11-15T04:33:38.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.120+0000 7fd9237fe700 1 -- 172.21.15.190:0/1293842018 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd930013f00 con 0x7fd93c0f5410 2023-11-15T04:33:38.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.120+0000 7fd9237fe700 1 -- 172.21.15.190:0/1293842018 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd930014d90 con 0x7fd93c0f5410 2023-11-15T04:33:38.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.120+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c0fa030 con 0x7fd93c0f5410 2023-11-15T04:33:38.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.121+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c059100 con 0x7fd93c0f5410 2023-11-15T04:33:38.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.121+0000 7fd9237fe700 1 -- 172.21.15.190:0/1293842018 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9300095f0 con 0x7fd93c0f5410 2023-11-15T04:33:38.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.121+0000 7fd9237fe700 1 --2- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd92402ff60 0x7fd924032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:38.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.121+0000 7fd9237fe700 1 -- 172.21.15.190:0/1293842018 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd930040fc0 con 0x7fd93c0f5410 2023-11-15T04:33:38.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.121+0000 7fd93a7fc700 1 --2- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd92402ff60 0x7fd924032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:38.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.122+0000 7fd93a7fc700 1 --2- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd92402ff60 0x7fd924032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fd92c0096a0 tx=0x7fd92c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:38.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.126+0000 7fd9237fe700 1 -- 172.21.15.190:0/1293842018 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd930014420 con 0x7fd93c0f5410 2023-11-15T04:33:38.331 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.331+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd93c045ad0 con 0x7fd93c0f5410 2023-11-15T04:33:38.331 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.331+0000 7fd9237fe700 1 -- 172.21.15.190:0/1293842018 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd930010230 con 0x7fd93c0f5410 2023-11-15T04:33:38.331 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:38.332 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:38.332 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd92402ff60 msgr2=0x7fd924032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:38.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 --2- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd92402ff60 0x7fd924032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fd92c0096a0 tx=0x7fd92c006b40).stop 2023-11-15T04:33:38.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 msgr2=0x7fd93c0fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:38.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 --2- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 0x7fd93c0fb580 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fd930013510 tx=0x7fd93000b040).stop 2023-11-15T04:33:38.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 shutdown_connections 2023-11-15T04:33:38.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 --2- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd92402ff60 0x7fd924032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:38.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 --2- 172.21.15.190:0/1293842018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd93c0f5410 0x7fd93c0fb580 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:38.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 >> 172.21.15.190:0/1293842018 conn(0x7fd93c0f0b70 msgr2=0x7fd93c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:38.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 shutdown_connections 2023-11-15T04:33:38.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:38.333+0000 7fd94171b700 1 -- 172.21.15.190:0/1293842018 wait complete. 2023-11-15T04:33:38.335 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:39.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:38 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:38.331807+0000 mon.smithi047 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.190:0/1293842018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:39.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:39.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:40.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:39 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:39.099418+0000 mgr.smithi047.zgysuz (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:40.999 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:40.999+0000 7fb08c81a700 1 -- 172.21.15.190:0/461584916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 msgr2=0x7fb0880f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:41.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:40.999+0000 7fb08c81a700 1 --2- 172.21.15.190:0/461584916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 0x7fb0880f8f20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fb078009fb0 tx=0x7fb078009b30).stop 2023-11-15T04:33:41.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:40.999+0000 7fb08c81a700 1 -- 172.21.15.190:0/461584916 shutdown_connections 2023-11-15T04:33:41.000 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:40.999+0000 7fb08c81a700 1 --2- 172.21.15.190:0/461584916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 0x7fb0880f8f20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:41.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:40.999+0000 7fb08c81a700 1 -- 172.21.15.190:0/461584916 >> 172.21.15.190:0/461584916 conn(0x7fb0880f09a0 msgr2=0x7fb0880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:41.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:40.999+0000 7fb08c81a700 1 -- 172.21.15.190:0/461584916 shutdown_connections 2023-11-15T04:33:41.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:40.999+0000 7fb08c81a700 1 -- 172.21.15.190:0/461584916 wait complete. 2023-11-15T04:33:41.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:40.999+0000 7fb08c81a700 1 Processor -- start 2023-11-15T04:33:41.001 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.000+0000 7fb08c81a700 1 -- start start 2023-11-15T04:33:41.002 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.000+0000 7fb08c81a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 0x7fb088106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:41.002 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.000+0000 7fb08c81a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0881069f0 con 0x7fb0880601e0 2023-11-15T04:33:41.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.000+0000 7fb0877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 0x7fb088106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:41.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.000+0000 7fb0877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 0x7fb088106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60326/0 (socket says 172.21.15.190:60326) 2023-11-15T04:33:41.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.000+0000 7fb0877fe700 1 -- 172.21.15.190:0/1280056684 learned_addr learned my addr 172.21.15.190:0/1280056684 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:41.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.000+0000 7fb0877fe700 1 -- 172.21.15.190:0/1280056684 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb078005040 con 0x7fb0880601e0 2023-11-15T04:33:41.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.000+0000 7fb0877fe700 1 --2- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 0x7fb088106500 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fb078013510 tx=0x7fb07800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:41.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.001+0000 7fb06ffff700 1 -- 172.21.15.190:0/1280056684 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb078009320 con 0x7fb0880601e0 2023-11-15T04:33:41.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.001+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb088106bf0 con 0x7fb0880601e0 2023-11-15T04:33:41.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.001+0000 7fb06ffff700 1 -- 172.21.15.190:0/1280056684 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb078013f00 con 0x7fb0880601e0 2023-11-15T04:33:41.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.001+0000 7fb06ffff700 1 -- 172.21.15.190:0/1280056684 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb078014d90 con 0x7fb0880601e0 2023-11-15T04:33:41.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.001+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb088107080 con 0x7fb0880601e0 2023-11-15T04:33:41.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.001+0000 7fb06ffff700 1 -- 172.21.15.190:0/1280056684 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb078014420 con 0x7fb0880601e0 2023-11-15T04:33:41.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.001+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb088045ad0 con 0x7fb0880601e0 2023-11-15T04:33:41.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.002+0000 7fb06ffff700 1 --2- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb07002ff60 0x7fb070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:41.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.002+0000 7fb06ffff700 1 -- 172.21.15.190:0/1280056684 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0780410c0 con 0x7fb0880601e0 2023-11-15T04:33:41.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.002+0000 7fb086ffd700 1 --2- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb07002ff60 0x7fb070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:41.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.002+0000 7fb086ffd700 1 --2- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb07002ff60 0x7fb070032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fb07c0096a0 tx=0x7fb07c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:41.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.007+0000 7fb06ffff700 1 -- 172.21.15.190:0/1280056684 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb078026bf0 con 0x7fb0880601e0 2023-11-15T04:33:41.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.213+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb088107f30 con 0x7fb0880601e0 2023-11-15T04:33:41.214 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.214+0000 7fb06ffff700 1 -- 172.21.15.190:0/1280056684 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb07801db70 con 0x7fb0880601e0 2023-11-15T04:33:41.215 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:41.215 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:41.216 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb07002ff60 msgr2=0x7fb070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:41.216 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 --2- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb07002ff60 0x7fb070032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fb07c0096a0 tx=0x7fb07c006b40).stop 2023-11-15T04:33:41.216 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 msgr2=0x7fb088106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:41.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 --2- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 0x7fb088106500 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fb078013510 tx=0x7fb07800b040).stop 2023-11-15T04:33:41.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 shutdown_connections 2023-11-15T04:33:41.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 --2- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb07002ff60 0x7fb070032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:41.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 --2- 172.21.15.190:0/1280056684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0880601e0 0x7fb088106500 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:41.217 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 >> 172.21.15.190:0/1280056684 conn(0x7fb0880f09a0 msgr2=0x7fb0880f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:41.218 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 shutdown_connections 2023-11-15T04:33:41.218 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:41.216+0000 7fb08c81a700 1 -- 172.21.15.190:0/1280056684 wait complete. 2023-11-15T04:33:41.218 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:42.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:41.099712+0000 mgr.smithi047.zgysuz (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:42.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:41.214737+0000 mon.smithi047 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.190:0/1280056684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:42.599 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:42.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:43.741 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.741+0000 7f555f48a700 1 -- 172.21.15.190:0/1384142350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 msgr2=0x7f55580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:43.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.741+0000 7f555f48a700 1 --2- 172.21.15.190:0/1384142350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55580f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f5548009fb0 tx=0x7f5548009b30).stop 2023-11-15T04:33:43.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.741+0000 7f555f48a700 1 -- 172.21.15.190:0/1384142350 shutdown_connections 2023-11-15T04:33:43.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.741+0000 7f555f48a700 1 --2- 172.21.15.190:0/1384142350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55580f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:43.743 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.741+0000 7f555f48a700 1 -- 172.21.15.190:0/1384142350 >> 172.21.15.190:0/1384142350 conn(0x7f55580f0b70 msgr2=0x7f55580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:43.743 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.741+0000 7f555f48a700 1 -- 172.21.15.190:0/1384142350 shutdown_connections 2023-11-15T04:33:43.743 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.741+0000 7f555f48a700 1 -- 172.21.15.190:0/1384142350 wait complete. 2023-11-15T04:33:43.743 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.742+0000 7f555f48a700 1 Processor -- start 2023-11-15T04:33:43.743 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.742+0000 7f555f48a700 1 -- start start 2023-11-15T04:33:43.744 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.742+0000 7f555f48a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f5558105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:43.744 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.742+0000 7f555f48a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5558105b20 con 0x7f55580f5410 2023-11-15T04:33:43.746 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.742+0000 7f555d226700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f5558105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:43.746 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.742+0000 7f555d226700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f5558105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60344/0 (socket says 172.21.15.190:60344) 2023-11-15T04:33:43.747 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.742+0000 7f555d226700 1 -- 172.21.15.190:0/4142825276 learned_addr learned my addr 172.21.15.190:0/4142825276 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:43.747 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.743+0000 7f555d226700 1 -- 172.21.15.190:0/4142825276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5548005040 con 0x7f55580f5410 2023-11-15T04:33:43.747 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.743+0000 7f555d226700 1 --2- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f5558105630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f5548013510 tx=0x7f554800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:43.748 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.743+0000 7f554dffb700 1 -- 172.21.15.190:0/4142825276 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5548009320 con 0x7f55580f5410 2023-11-15T04:33:43.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.743+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5558105d20 con 0x7f55580f5410 2023-11-15T04:33:43.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.743+0000 7f554dffb700 1 -- 172.21.15.190:0/4142825276 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5548013f00 con 0x7f55580f5410 2023-11-15T04:33:43.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.743+0000 7f554dffb700 1 -- 172.21.15.190:0/4142825276 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5548014d90 con 0x7f55580f5410 2023-11-15T04:33:43.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.743+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55581061b0 con 0x7f55580f5410 2023-11-15T04:33:43.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.744+0000 7f554dffb700 1 -- 172.21.15.190:0/4142825276 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55480095f0 con 0x7f55580f5410 2023-11-15T04:33:43.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.744+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5558045ad0 con 0x7f55580f5410 2023-11-15T04:33:43.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.744+0000 7f554dffb700 1 --2- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f554402ff60 0x7f5544032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:43.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.744+0000 7f554dffb700 1 -- 172.21.15.190:0/4142825276 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5548042d90 con 0x7f55580f5410 2023-11-15T04:33:43.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.744+0000 7f555ca25700 1 --2- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f554402ff60 0x7f5544032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:43.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.745+0000 7f555ca25700 1 --2- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f554402ff60 0x7f5544032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f55540096a0 tx=0x7f5554006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:43.755 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.749+0000 7f554dffb700 1 -- 172.21.15.190:0/4142825276 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5548014420 con 0x7f55580f5410 2023-11-15T04:33:43.957 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.957+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55580fae60 con 0x7f55580f5410 2023-11-15T04:33:43.958 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.958+0000 7f554dffb700 1 -- 172.21.15.190:0/4142825276 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5548011290 con 0x7f55580f5410 2023-11-15T04:33:43.958 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:43.958 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f554402ff60 msgr2=0x7f5544032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 --2- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f554402ff60 0x7f5544032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f55540096a0 tx=0x7f5554006b40).stop 2023-11-15T04:33:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 msgr2=0x7f5558105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 --2- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f5558105630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f5548013510 tx=0x7f554800b040).stop 2023-11-15T04:33:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 shutdown_connections 2023-11-15T04:33:43.961 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 --2- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f554402ff60 0x7f5544032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:43.961 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 --2- 172.21.15.190:0/4142825276 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f5558105630 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:43.961 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 >> 172.21.15.190:0/4142825276 conn(0x7f55580f0b70 msgr2=0x7f55580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:43.961 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 shutdown_connections 2023-11-15T04:33:43.961 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:43.960+0000 7f555f48a700 1 -- 172.21.15.190:0/4142825276 wait complete. 2023-11-15T04:33:43.962 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:43.100000+0000 mgr.smithi047.zgysuz (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:43.958423+0000 mon.smithi047 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.190:0/4142825276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:45.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:45.339 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:46.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.482+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/3094683961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 msgr2=0x7fd3bc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:46.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.482+0000 7fd3c2b3b700 1 --2- 172.21.15.190:0/3094683961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 0x7fd3bc0f66c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fd3ac009fb0 tx=0x7fd3ac009b30).stop 2023-11-15T04:33:46.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.482+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/3094683961 shutdown_connections 2023-11-15T04:33:46.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.482+0000 7fd3c2b3b700 1 --2- 172.21.15.190:0/3094683961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 0x7fd3bc0f66c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:46.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.482+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/3094683961 >> 172.21.15.190:0/3094683961 conn(0x7fd3bc0f1aa0 msgr2=0x7fd3bc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:46.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.482+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/3094683961 shutdown_connections 2023-11-15T04:33:46.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.482+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/3094683961 wait complete. 2023-11-15T04:33:46.484 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.482+0000 7fd3c2b3b700 1 Processor -- start 2023-11-15T04:33:46.485 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.483+0000 7fd3c2b3b700 1 -- start start 2023-11-15T04:33:46.485 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.483+0000 7fd3c2b3b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 0x7fd3bc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:46.485 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.483+0000 7fd3c2b3b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc1069a0 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.486 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.483+0000 7fd3c1b39700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 0x7fd3bc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:46.487 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.483+0000 7fd3c1b39700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 0x7fd3bc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60360/0 (socket says 172.21.15.190:60360) 2023-11-15T04:33:46.487 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.483+0000 7fd3c1b39700 1 -- 172.21.15.190:0/511201739 learned_addr learned my addr 172.21.15.190:0/511201739 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:46.488 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.483+0000 7fd3c1b39700 1 -- 172.21.15.190:0/511201739 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac005040 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.488 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.483+0000 7fd3c1b39700 1 --2- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 0x7fd3bc1064b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fd3ac013510 tx=0x7fd3ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:46.489 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.484+0000 7fd3b27fc700 1 -- 172.21.15.190:0/511201739 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3ac009320 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.489 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.484+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3bc106ba0 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.489 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.484+0000 7fd3b27fc700 1 -- 172.21.15.190:0/511201739 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3ac013f00 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.489 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.484+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3bc107030 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.490 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.484+0000 7fd3b27fc700 1 -- 172.21.15.190:0/511201739 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3ac014dc0 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.490 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.484+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3bc045ad0 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.490 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.484+0000 7fd3b27fc700 1 -- 172.21.15.190:0/511201739 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3ac0095f0 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.490 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.485+0000 7fd3b27fc700 1 --2- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd3a802ff60 0x7fd3a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:46.490 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.485+0000 7fd3b27fc700 1 -- 172.21.15.190:0/511201739 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3ac041000 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.485+0000 7fd3c1338700 1 --2- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd3a802ff60 0x7fd3a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:46.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.485+0000 7fd3c1338700 1 --2- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd3a802ff60 0x7fd3a8032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd3b80096a0 tx=0x7fd3b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:46.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.489+0000 7fd3b27fc700 1 -- 172.21.15.190:0/511201739 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3ac01c790 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:45.100361+0000 mgr.smithi047.zgysuz (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:46.697 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.697+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3bc107e80 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.697 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.698+0000 7fd3b27fc700 1 -- 172.21.15.190:0/511201739 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3ac01c790 con 0x7fd3bc0f62a0 2023-11-15T04:33:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:46.698 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:46.699 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd3a802ff60 msgr2=0x7fd3a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:46.699 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 --2- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd3a802ff60 0x7fd3a8032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd3b80096a0 tx=0x7fd3b8006b40).stop 2023-11-15T04:33:46.699 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 msgr2=0x7fd3bc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:46.700 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 --2- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 0x7fd3bc1064b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fd3ac013510 tx=0x7fd3ac00b040).stop 2023-11-15T04:33:46.700 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 shutdown_connections 2023-11-15T04:33:46.700 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 --2- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd3a802ff60 0x7fd3a8032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:46.700 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 --2- 172.21.15.190:0/511201739 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3bc0f62a0 0x7fd3bc1064b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:46.700 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 >> 172.21.15.190:0/511201739 conn(0x7fd3bc0f1aa0 msgr2=0x7fd3bc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:46.701 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 shutdown_connections 2023-11-15T04:33:46.701 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:46.699+0000 7fd3c2b3b700 1 -- 172.21.15.190:0/511201739 wait complete. 2023-11-15T04:33:46.701 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:47.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:46.698267+0000 mon.smithi047 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.190:0/511201739' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:48.115 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:48.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:48.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:48 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:47.100663+0000 mgr.smithi047.zgysuz (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:49.252 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.251+0000 7ff20d612700 1 -- 172.21.15.190:0/3622367938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 msgr2=0x7ff2080f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:49.252 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.251+0000 7ff20d612700 1 --2- 172.21.15.190:0/3622367938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 0x7ff2080f5750 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7ff1fc009fb0 tx=0x7ff1fc009b30).stop 2023-11-15T04:33:49.252 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.251+0000 7ff20d612700 1 -- 172.21.15.190:0/3622367938 shutdown_connections 2023-11-15T04:33:49.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.251+0000 7ff20d612700 1 --2- 172.21.15.190:0/3622367938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 0x7ff2080f5750 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:49.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.251+0000 7ff20d612700 1 -- 172.21.15.190:0/3622367938 >> 172.21.15.190:0/3622367938 conn(0x7ff2080f0a90 msgr2=0x7ff2080f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:49.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.251+0000 7ff20d612700 1 -- 172.21.15.190:0/3622367938 shutdown_connections 2023-11-15T04:33:49.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.251+0000 7ff20d612700 1 -- 172.21.15.190:0/3622367938 wait complete. 2023-11-15T04:33:49.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.252+0000 7ff20d612700 1 Processor -- start 2023-11-15T04:33:49.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.252+0000 7ff20d612700 1 -- start start 2023-11-15T04:33:49.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.252+0000 7ff20d612700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 0x7ff2081096f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:49.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.252+0000 7ff20d612700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff208060220 con 0x7ff2080f5330 2023-11-15T04:33:49.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.252+0000 7ff206ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 0x7ff2081096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:49.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.252+0000 7ff206ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 0x7ff2081096f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60364/0 (socket says 172.21.15.190:60364) 2023-11-15T04:33:49.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.252+0000 7ff206ffd700 1 -- 172.21.15.190:0/3938880748 learned_addr learned my addr 172.21.15.190:0/3938880748 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:49.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.252+0000 7ff206ffd700 1 -- 172.21.15.190:0/3938880748 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc005040 con 0x7ff2080f5330 2023-11-15T04:33:49.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.252+0000 7ff206ffd700 1 --2- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 0x7ff2081096f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff1fc013510 tx=0x7ff1fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:49.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.253+0000 7ff1ef7fe700 1 -- 172.21.15.190:0/3938880748 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1fc009320 con 0x7ff2080f5330 2023-11-15T04:33:49.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.253+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff208060420 con 0x7ff2080f5330 2023-11-15T04:33:49.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.253+0000 7ff1ef7fe700 1 -- 172.21.15.190:0/3938880748 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1fc013f00 con 0x7ff2080f5330 2023-11-15T04:33:49.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.253+0000 7ff1ef7fe700 1 -- 172.21.15.190:0/3938880748 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1fc014d90 con 0x7ff2080f5330 2023-11-15T04:33:49.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.253+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff208109da0 con 0x7ff2080f5330 2023-11-15T04:33:49.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.253+0000 7ff1ef7fe700 1 -- 172.21.15.190:0/3938880748 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1fc014420 con 0x7ff2080f5330 2023-11-15T04:33:49.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.253+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff208045ad0 con 0x7ff2080f5330 2023-11-15T04:33:49.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.253+0000 7ff1ef7fe700 1 --2- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1f002ff60 0x7ff1f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:49.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.254+0000 7ff1ef7fe700 1 -- 172.21.15.190:0/3938880748 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1fc0410d0 con 0x7ff2080f5330 2023-11-15T04:33:49.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.254+0000 7ff2067fc700 1 --2- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1f002ff60 0x7ff1f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:49.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.254+0000 7ff2067fc700 1 --2- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1f002ff60 0x7ff1f0032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ff1f80096a0 tx=0x7ff1f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:49.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.259+0000 7ff1ef7fe700 1 -- 172.21.15.190:0/3938880748 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1fc0095f0 con 0x7ff2080f5330 2023-11-15T04:33:49.465 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.465+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff20805ea50 con 0x7ff2080f5330 2023-11-15T04:33:49.466 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.466+0000 7ff1ef7fe700 1 -- 172.21.15.190:0/3938880748 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff1fc01db70 con 0x7ff2080f5330 2023-11-15T04:33:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:49.467 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:49.467 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.467+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1f002ff60 msgr2=0x7ff1f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:49.468 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.467+0000 7ff20d612700 1 --2- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1f002ff60 0x7ff1f0032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ff1f80096a0 tx=0x7ff1f8006b40).stop 2023-11-15T04:33:49.468 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.467+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 msgr2=0x7ff2081096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:49.468 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.467+0000 7ff20d612700 1 --2- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 0x7ff2081096f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff1fc013510 tx=0x7ff1fc00b040).stop 2023-11-15T04:33:49.468 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.468+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 shutdown_connections 2023-11-15T04:33:49.468 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.468+0000 7ff20d612700 1 --2- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1f002ff60 0x7ff1f0032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:49.469 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.468+0000 7ff20d612700 1 --2- 172.21.15.190:0/3938880748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2080f5330 0x7ff2081096f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:49.469 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.468+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 >> 172.21.15.190:0/3938880748 conn(0x7ff2080f0a90 msgr2=0x7ff2080f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:49.469 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.468+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 shutdown_connections 2023-11-15T04:33:49.470 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:49.468+0000 7ff20d612700 1 -- 172.21.15.190:0/3938880748 wait complete. 2023-11-15T04:33:49.470 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:50.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:50 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:49.101033+0000 mgr.smithi047.zgysuz (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:50.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:50 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:49.466536+0000 mon.smithi047 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.190:0/3938880748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:50.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:50.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:51.963 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.963+0000 7fe13aa77700 1 -- 172.21.15.190:0/2013241138 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 msgr2=0x7fe1340f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:51.964 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.963+0000 7fe13aa77700 1 --2- 172.21.15.190:0/2013241138 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 0x7fe1340f6e20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fe12c009960 tx=0x7fe12c004d10).stop 2023-11-15T04:33:51.964 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.963+0000 7fe13aa77700 1 -- 172.21.15.190:0/2013241138 shutdown_connections 2023-11-15T04:33:51.964 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.963+0000 7fe13aa77700 1 --2- 172.21.15.190:0/2013241138 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 0x7fe1340f6e20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:51.964 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.963+0000 7fe13aa77700 1 -- 172.21.15.190:0/2013241138 >> 172.21.15.190:0/2013241138 conn(0x7fe1340f0260 msgr2=0x7fe1340f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:51.965 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.963+0000 7fe13aa77700 1 -- 172.21.15.190:0/2013241138 shutdown_connections 2023-11-15T04:33:51.965 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.963+0000 7fe13aa77700 1 -- 172.21.15.190:0/2013241138 wait complete. 2023-11-15T04:33:51.965 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.964+0000 7fe13aa77700 1 Processor -- start 2023-11-15T04:33:51.965 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.964+0000 7fe13aa77700 1 -- start start 2023-11-15T04:33:51.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.964+0000 7fe13aa77700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 0x7fe13410b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:51.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.964+0000 7fe13aa77700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1341071c0 con 0x7fe1340f6a00 2023-11-15T04:33:51.967 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.964+0000 7fe138813700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 0x7fe13410b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:51.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.964+0000 7fe138813700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 0x7fe13410b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49600/0 (socket says 172.21.15.190:49600) 2023-11-15T04:33:51.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.964+0000 7fe138813700 1 -- 172.21.15.190:0/1585895266 learned_addr learned my addr 172.21.15.190:0/1585895266 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:51.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.965+0000 7fe138813700 1 -- 172.21.15.190:0/1585895266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12c009640 con 0x7fe1340f6a00 2023-11-15T04:33:51.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.965+0000 7fe138813700 1 --2- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 0x7fe13410b7d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fe12c007bd0 tx=0x7fe12c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:51.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.965+0000 7fe1317fa700 1 -- 172.21.15.190:0/1585895266 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe12c018070 con 0x7fe1340f6a00 2023-11-15T04:33:51.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.965+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1340f5290 con 0x7fe1340f6a00 2023-11-15T04:33:51.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.965+0000 7fe1317fa700 1 -- 172.21.15.190:0/1585895266 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe12c00b870 con 0x7fe1340f6a00 2023-11-15T04:33:51.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.965+0000 7fe1317fa700 1 -- 172.21.15.190:0/1585895266 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe12c01d420 con 0x7fe1340f6a00 2023-11-15T04:33:51.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.965+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe13410bcc0 con 0x7fe1340f6a00 2023-11-15T04:33:51.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.966+0000 7fe1317fa700 1 -- 172.21.15.190:0/1585895266 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe12c024060 con 0x7fe1340f6a00 2023-11-15T04:33:51.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.966+0000 7fe1317fa700 1 --2- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe11c02ff60 0x7fe11c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:51.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.966+0000 7fe1317fa700 1 -- 172.21.15.190:0/1585895266 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe12c041180 con 0x7fe1340f6a00 2023-11-15T04:33:51.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.966+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe134045ad0 con 0x7fe1340f6a00 2023-11-15T04:33:51.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.966+0000 7fe133fff700 1 --2- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe11c02ff60 0x7fe11c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:51.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.967+0000 7fe133fff700 1 --2- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe11c02ff60 0x7fe11c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe1280096a0 tx=0x7fe128006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:51.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:51.971+0000 7fe1317fa700 1 -- 172.21.15.190:0/1585895266 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe12c025340 con 0x7fe1340f6a00 2023-11-15T04:33:52.177 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.177+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1340fafe0 con 0x7fe1340f6a00 2023-11-15T04:33:52.177 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.178+0000 7fe1317fa700 1 -- 172.21.15.190:0/1585895266 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe12c011b10 con 0x7fe1340f6a00 2023-11-15T04:33:52.178 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:52.178 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:52.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe11c02ff60 msgr2=0x7fe11c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:52.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 --2- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe11c02ff60 0x7fe11c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe1280096a0 tx=0x7fe128006b40).stop 2023-11-15T04:33:52.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 msgr2=0x7fe13410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:52.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 --2- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 0x7fe13410b7d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fe12c007bd0 tx=0x7fe12c004d10).stop 2023-11-15T04:33:52.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 shutdown_connections 2023-11-15T04:33:52.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 --2- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe11c02ff60 0x7fe11c032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:52.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 --2- 172.21.15.190:0/1585895266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1340f6a00 0x7fe13410b7d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:52.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 >> 172.21.15.190:0/1585895266 conn(0x7fe1340f0260 msgr2=0x7fe1340f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:52.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 shutdown_connections 2023-11-15T04:33:52.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:52.179+0000 7fe13aa77700 1 -- 172.21.15.190:0/1585895266 wait complete. 2023-11-15T04:33:52.182 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:51.101409+0000 mgr.smithi047.zgysuz (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:53.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:52.178350+0000 mon.smithi047 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.190:0/1585895266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:53.554 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:53.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:53.101672+0000 mgr.smithi047.zgysuz (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:54.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.756+0000 7faca6f94700 1 -- 172.21.15.190:0/544927565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 msgr2=0x7faca00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:54.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.756+0000 7faca6f94700 1 --2- 172.21.15.190:0/544927565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 0x7faca00f6930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fac94009fb0 tx=0x7fac94009b30).stop 2023-11-15T04:33:54.757 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.756+0000 7faca6f94700 1 -- 172.21.15.190:0/544927565 shutdown_connections 2023-11-15T04:33:54.758 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.756+0000 7faca6f94700 1 --2- 172.21.15.190:0/544927565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 0x7faca00f6930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:54.758 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.756+0000 7faca6f94700 1 -- 172.21.15.190:0/544927565 >> 172.21.15.190:0/544927565 conn(0x7faca00f1c70 msgr2=0x7faca00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:54.758 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.756+0000 7faca6f94700 1 -- 172.21.15.190:0/544927565 shutdown_connections 2023-11-15T04:33:54.758 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.756+0000 7faca6f94700 1 -- 172.21.15.190:0/544927565 wait complete. 2023-11-15T04:33:54.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.756+0000 7faca6f94700 1 Processor -- start 2023-11-15T04:33:54.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.756+0000 7faca6f94700 1 -- start start 2023-11-15T04:33:54.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.757+0000 7faca6f94700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 0x7faca006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:54.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.757+0000 7faca6f94700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca0069e20 con 0x7faca00f6510 2023-11-15T04:33:54.762 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.757+0000 7faca4d30700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 0x7faca006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:54.762 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.757+0000 7faca4d30700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 0x7faca006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49610/0 (socket says 172.21.15.190:49610) 2023-11-15T04:33:54.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.757+0000 7faca4d30700 1 -- 172.21.15.190:0/3321627434 learned_addr learned my addr 172.21.15.190:0/3321627434 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:54.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.758+0000 7faca4d30700 1 -- 172.21.15.190:0/3321627434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac94005040 con 0x7faca00f6510 2023-11-15T04:33:54.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.758+0000 7faca4d30700 1 --2- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 0x7faca006b9a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fac94013510 tx=0x7fac9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:54.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.758+0000 7fac9d7fa700 1 -- 172.21.15.190:0/3321627434 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac94009320 con 0x7faca00f6510 2023-11-15T04:33:54.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.758+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca0069fc0 con 0x7faca00f6510 2023-11-15T04:33:54.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.758+0000 7fac9d7fa700 1 -- 172.21.15.190:0/3321627434 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac94013f00 con 0x7faca00f6510 2023-11-15T04:33:54.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.758+0000 7fac9d7fa700 1 -- 172.21.15.190:0/3321627434 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac94014d90 con 0x7faca00f6510 2023-11-15T04:33:54.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.758+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca006a450 con 0x7faca00f6510 2023-11-15T04:33:54.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.759+0000 7fac9d7fa700 1 -- 172.21.15.190:0/3321627434 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac94014420 con 0x7faca00f6510 2023-11-15T04:33:54.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.759+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca0059100 con 0x7faca00f6510 2023-11-15T04:33:54.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.759+0000 7fac9d7fa700 1 --2- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fac9002ff60 0x7fac90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:54.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.759+0000 7fac9d7fa700 1 -- 172.21.15.190:0/3321627434 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac94041000 con 0x7faca00f6510 2023-11-15T04:33:54.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.759+0000 7fac9ffff700 1 --2- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fac9002ff60 0x7fac90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:54.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.760+0000 7fac9ffff700 1 --2- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fac9002ff60 0x7fac90032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fac8c0096a0 tx=0x7fac8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:54.766 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.764+0000 7fac9d7fa700 1 -- 172.21.15.190:0/3321627434 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac940095f0 con 0x7faca00f6510 2023-11-15T04:33:54.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.968+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faca00fbf60 con 0x7faca00f6510 2023-11-15T04:33:54.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.968+0000 7fac9d7fa700 1 -- 172.21.15.190:0/3321627434 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac9401db70 con 0x7faca00f6510 2023-11-15T04:33:54.970 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:54.970 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:54.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fac9002ff60 msgr2=0x7fac90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:54.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 --2- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fac9002ff60 0x7fac90032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fac8c0096a0 tx=0x7fac8c006b40).stop 2023-11-15T04:33:54.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 msgr2=0x7faca006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:54.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 --2- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 0x7faca006b9a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fac94013510 tx=0x7fac9400b040).stop 2023-11-15T04:33:54.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 shutdown_connections 2023-11-15T04:33:54.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 --2- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fac9002ff60 0x7fac90032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:54.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 --2- 172.21.15.190:0/3321627434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faca00f6510 0x7faca006b9a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:54.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 >> 172.21.15.190:0/3321627434 conn(0x7faca00f1c70 msgr2=0x7faca00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:54.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 shutdown_connections 2023-11-15T04:33:54.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:54.970+0000 7faca6f94700 1 -- 172.21.15.190:0/3321627434 wait complete. 2023-11-15T04:33:54.973 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:55.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:55 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:54.969829+0000 mon.smithi047 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.190:0/3321627434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:56.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:56.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:33:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:55.102062+0000 mgr.smithi047.zgysuz (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:57.490 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.489+0000 7fd9845a3700 1 -- 172.21.15.190:0/2481024259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 msgr2=0x7fd97c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:57.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.489+0000 7fd9845a3700 1 --2- 172.21.15.190:0/2481024259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 0x7fd97c0f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fd970009fb0 tx=0x7fd970009b30).stop 2023-11-15T04:33:57.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.489+0000 7fd9845a3700 1 -- 172.21.15.190:0/2481024259 shutdown_connections 2023-11-15T04:33:57.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.489+0000 7fd9845a3700 1 --2- 172.21.15.190:0/2481024259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 0x7fd97c0f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:57.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.489+0000 7fd9845a3700 1 -- 172.21.15.190:0/2481024259 >> 172.21.15.190:0/2481024259 conn(0x7fd97c0f0b70 msgr2=0x7fd97c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:57.491 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.489+0000 7fd9845a3700 1 -- 172.21.15.190:0/2481024259 shutdown_connections 2023-11-15T04:33:57.492 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.489+0000 7fd9845a3700 1 -- 172.21.15.190:0/2481024259 wait complete. 2023-11-15T04:33:57.493 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.490+0000 7fd9845a3700 1 Processor -- start 2023-11-15T04:33:57.494 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.490+0000 7fd9845a3700 1 -- start start 2023-11-15T04:33:57.494 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.490+0000 7fd9845a3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 0x7fd97c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:57.494 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.490+0000 7fd9845a3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c105b20 con 0x7fd97c0f5410 2023-11-15T04:33:57.494 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd98233f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 0x7fd97c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:57.495 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd98233f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 0x7fd97c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49628/0 (socket says 172.21.15.190:49628) 2023-11-15T04:33:57.495 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd98233f700 1 -- 172.21.15.190:0/2328254359 learned_addr learned my addr 172.21.15.190:0/2328254359 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:33:57.495 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd98233f700 1 -- 172.21.15.190:0/2328254359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd970005040 con 0x7fd97c0f5410 2023-11-15T04:33:57.495 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd98233f700 1 --2- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 0x7fd97c105630 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fd970013510 tx=0x7fd97000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:57.495 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd96effd700 1 -- 172.21.15.190:0/2328254359 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd970009320 con 0x7fd97c0f5410 2023-11-15T04:33:57.496 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c105d20 con 0x7fd97c0f5410 2023-11-15T04:33:57.496 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd96effd700 1 -- 172.21.15.190:0/2328254359 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd970013f00 con 0x7fd97c0f5410 2023-11-15T04:33:57.496 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd96effd700 1 -- 172.21.15.190:0/2328254359 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd970014d90 con 0x7fd97c0f5410 2023-11-15T04:33:57.496 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.491+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c1061b0 con 0x7fd97c0f5410 2023-11-15T04:33:57.497 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.492+0000 7fd96effd700 1 -- 172.21.15.190:0/2328254359 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9700095f0 con 0x7fd97c0f5410 2023-11-15T04:33:57.497 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.492+0000 7fd96effd700 1 --2- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd96802ff60 0x7fd968032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:33:57.497 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.492+0000 7fd96effd700 1 -- 172.21.15.190:0/2328254359 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9700410c0 con 0x7fd97c0f5410 2023-11-15T04:33:57.497 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.492+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c045ad0 con 0x7fd97c0f5410 2023-11-15T04:33:57.497 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.493+0000 7fd981b3e700 1 --2- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd96802ff60 0x7fd968032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:33:57.498 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.493+0000 7fd981b3e700 1 --2- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd96802ff60 0x7fd968032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fd9780096a0 tx=0x7fd978006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:33:57.498 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.497+0000 7fd96effd700 1 -- 172.21.15.190:0/2328254359 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd970014420 con 0x7fd97c0f5410 2023-11-15T04:33:57.706 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.704+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd97c0faf50 con 0x7fd97c0f5410 2023-11-15T04:33:57.706 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.705+0000 7fd96effd700 1 -- 172.21.15.190:0/2328254359 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd97002b020 con 0x7fd97c0f5410 2023-11-15T04:33:57.707 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:33:57.707 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:33:57.708 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd96802ff60 msgr2=0x7fd968032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:57.708 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 --2- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd96802ff60 0x7fd968032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fd9780096a0 tx=0x7fd978006b40).stop 2023-11-15T04:33:57.708 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 msgr2=0x7fd97c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:33:57.709 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 --2- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 0x7fd97c105630 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fd970013510 tx=0x7fd97000b040).stop 2023-11-15T04:33:57.709 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 shutdown_connections 2023-11-15T04:33:57.709 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 --2- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd96802ff60 0x7fd968032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:57.709 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 --2- 172.21.15.190:0/2328254359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd97c0f5410 0x7fd97c105630 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:33:57.709 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 >> 172.21.15.190:0/2328254359 conn(0x7fd97c0f0b70 msgr2=0x7fd97c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:33:57.710 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 shutdown_connections 2023-11-15T04:33:57.710 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:33:57.707+0000 7fd9845a3700 1 -- 172.21.15.190:0/2328254359 wait complete. 2023-11-15T04:33:57.710 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:33:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:57.102407+0000 mgr.smithi047.zgysuz (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:33:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:33:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:33:57.706557+0000 mon.smithi047 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.190:0/2328254359' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:33:59.100 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:33:59.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:00.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.263+0000 7f202a5f1700 1 -- 172.21.15.190:0/3981451859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 msgr2=0x7f20240f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:00.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.263+0000 7f202a5f1700 1 --2- 172.21.15.190:0/3981451859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 0x7f20240f5750 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f2014009fb0 tx=0x7f2014009b30).stop 2023-11-15T04:34:00.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.263+0000 7f202a5f1700 1 -- 172.21.15.190:0/3981451859 shutdown_connections 2023-11-15T04:34:00.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.263+0000 7f202a5f1700 1 --2- 172.21.15.190:0/3981451859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 0x7f20240f5750 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:00.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.263+0000 7f202a5f1700 1 -- 172.21.15.190:0/3981451859 >> 172.21.15.190:0/3981451859 conn(0x7f20240f0a90 msgr2=0x7f20240f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:00.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.263+0000 7f202a5f1700 1 -- 172.21.15.190:0/3981451859 shutdown_connections 2023-11-15T04:34:00.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.263+0000 7f202a5f1700 1 -- 172.21.15.190:0/3981451859 wait complete. 2023-11-15T04:34:00.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.263+0000 7f202a5f1700 1 Processor -- start 2023-11-15T04:34:00.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.264+0000 7f202a5f1700 1 -- start start 2023-11-15T04:34:00.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.264+0000 7f202a5f1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 0x7f20241058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.264+0000 7f202a5f1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2024105dc0 con 0x7f20240f5330 2023-11-15T04:34:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.264+0000 7f2023fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 0x7f20241058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.264+0000 7f2023fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 0x7f20241058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:44082/0 (socket says 172.21.15.190:44082) 2023-11-15T04:34:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.264+0000 7f2023fff700 1 -- 172.21.15.190:0/1755496877 learned_addr learned my addr 172.21.15.190:0/1755496877 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.264+0000 7f2023fff700 1 -- 172.21.15.190:0/1755496877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2014005040 con 0x7f20240f5330 2023-11-15T04:34:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.264+0000 7f2023fff700 1 --2- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 0x7f20241058d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2014013510 tx=0x7f201400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.265+0000 7f2020ff9700 1 -- 172.21.15.190:0/1755496877 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2014009320 con 0x7f20240f5330 2023-11-15T04:34:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.265+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2024105fc0 con 0x7f20240f5330 2023-11-15T04:34:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.265+0000 7f2020ff9700 1 -- 172.21.15.190:0/1755496877 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2014013f00 con 0x7f20240f5330 2023-11-15T04:34:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.265+0000 7f2020ff9700 1 -- 172.21.15.190:0/1755496877 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2014014d90 con 0x7f20240f5330 2023-11-15T04:34:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.265+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2024106450 con 0x7f20240f5330 2023-11-15T04:34:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.265+0000 7f2020ff9700 1 -- 172.21.15.190:0/1755496877 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20140263a0 con 0x7f20240f5330 2023-11-15T04:34:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.266+0000 7f2020ff9700 1 --2- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f201002ffb0 0x7f2010032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.266+0000 7f2020ff9700 1 -- 172.21.15.190:0/1755496877 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2014040ff0 con 0x7f20240f5330 2023-11-15T04:34:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.266+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2024045ad0 con 0x7f20240f5330 2023-11-15T04:34:00.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.266+0000 7f20237fe700 1 --2- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f201002ffb0 0x7f2010032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:00.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.266+0000 7f20237fe700 1 --2- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f201002ffb0 0x7f2010032470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f200c0096a0 tx=0x7f200c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:00.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.271+0000 7f2020ff9700 1 -- 172.21.15.190:0/1755496877 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2014012920 con 0x7f20240f5330 2023-11-15T04:34:00.476 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.476+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20241070d0 con 0x7f20240f5330 2023-11-15T04:34:00.477 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.476+0000 7f2020ff9700 1 -- 172.21.15.190:0/1755496877 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f201402b020 con 0x7f20240f5330 2023-11-15T04:34:00.477 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:00.478 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f201002ffb0 msgr2=0x7f2010032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 --2- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f201002ffb0 0x7f2010032470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f200c0096a0 tx=0x7f200c006b40).stop 2023-11-15T04:34:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 msgr2=0x7f20241058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 --2- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 0x7f20241058d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2014013510 tx=0x7f201400b040).stop 2023-11-15T04:34:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 shutdown_connections 2023-11-15T04:34:00.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 --2- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f201002ffb0 0x7f2010032470 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:00.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 --2- 172.21.15.190:0/1755496877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20240f5330 0x7f20241058d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:00.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 >> 172.21.15.190:0/1755496877 conn(0x7f20240f0a90 msgr2=0x7f20240f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:00.480 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 shutdown_connections 2023-11-15T04:34:00.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:00.478+0000 7f202a5f1700 1 -- 172.21.15.190:0/1755496877 wait complete. 2023-11-15T04:34:00.481 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:33:59.102799+0000 mgr.smithi047.zgysuz (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:01.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:01 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:00.477867+0000 mon.smithi047 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.190:0/1755496877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:01.860 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:01.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:01.103191+0000 mgr.smithi047.zgysuz (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:03.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.028+0000 7f2a2bfff700 1 -- 172.21.15.190:0/1876853934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 msgr2=0x7f2a2c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:03.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.028+0000 7f2a2bfff700 1 --2- 172.21.15.190:0/1876853934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 0x7f2a2c0f55c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f2a20009fb0 tx=0x7f2a20009b30).stop 2023-11-15T04:34:03.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.028+0000 7f2a2bfff700 1 -- 172.21.15.190:0/1876853934 shutdown_connections 2023-11-15T04:34:03.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.028+0000 7f2a2bfff700 1 --2- 172.21.15.190:0/1876853934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 0x7f2a2c0f55c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:03.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.028+0000 7f2a2bfff700 1 -- 172.21.15.190:0/1876853934 >> 172.21.15.190:0/1876853934 conn(0x7f2a2c0f09a0 msgr2=0x7f2a2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:03.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.028+0000 7f2a2bfff700 1 -- 172.21.15.190:0/1876853934 shutdown_connections 2023-11-15T04:34:03.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.028+0000 7f2a2bfff700 1 -- 172.21.15.190:0/1876853934 wait complete. 2023-11-15T04:34:03.033 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.029+0000 7f2a2bfff700 1 Processor -- start 2023-11-15T04:34:03.033 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.029+0000 7f2a2bfff700 1 -- start start 2023-11-15T04:34:03.033 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.029+0000 7f2a2bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 0x7f2a2c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:03.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.029+0000 7f2a2bfff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c1058a0 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.029+0000 7f2a2affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 0x7f2a2c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:03.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.029+0000 7f2a2affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 0x7f2a2c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:44102/0 (socket says 172.21.15.190:44102) 2023-11-15T04:34:03.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.029+0000 7f2a2affd700 1 -- 172.21.15.190:0/2007062097 learned_addr learned my addr 172.21.15.190:0/2007062097 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:03.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.030+0000 7f2a2affd700 1 -- 172.21.15.190:0/2007062097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a20005040 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.035 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.030+0000 7f2a2affd700 1 --2- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 0x7f2a2c1053b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f2a20013510 tx=0x7f2a2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:03.035 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.030+0000 7f2a137fe700 1 -- 172.21.15.190:0/2007062097 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a20009320 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.035 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.030+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c105aa0 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.035 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.030+0000 7f2a137fe700 1 -- 172.21.15.190:0/2007062097 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a20013f00 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.035 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.030+0000 7f2a137fe700 1 -- 172.21.15.190:0/2007062097 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a20014dc0 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.036 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.030+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c105f30 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.036 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.031+0000 7f2a137fe700 1 -- 172.21.15.190:0/2007062097 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a20014420 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.036 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.031+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c045ad0 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.036 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.031+0000 7f2a137fe700 1 --2- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2a1402ff60 0x7f2a14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:03.036 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.031+0000 7f2a137fe700 1 -- 172.21.15.190:0/2007062097 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a200410d0 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.037 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.031+0000 7f2a2a7fc700 1 --2- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2a1402ff60 0x7f2a14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:03.037 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.032+0000 7f2a2a7fc700 1 --2- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2a1402ff60 0x7f2a14032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2a1c0096a0 tx=0x7f2a1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:03.038 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.036+0000 7f2a137fe700 1 -- 172.21.15.190:0/2007062097 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a200095f0 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.240 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.240+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a2c106de0 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.241 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.240+0000 7f2a137fe700 1 -- 172.21.15.190:0/2007062097 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a2001cb10 con 0x7f2a2c0f51a0 2023-11-15T04:34:03.242 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:03.242 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:03.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2a1402ff60 msgr2=0x7f2a14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:03.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 --2- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2a1402ff60 0x7f2a14032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2a1c0096a0 tx=0x7f2a1c006b40).stop 2023-11-15T04:34:03.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 msgr2=0x7f2a2c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:03.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 --2- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 0x7f2a2c1053b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f2a20013510 tx=0x7f2a2000b040).stop 2023-11-15T04:34:03.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 shutdown_connections 2023-11-15T04:34:03.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 --2- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2a1402ff60 0x7f2a14032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:03.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 --2- 172.21.15.190:0/2007062097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a2c0f51a0 0x7f2a2c1053b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:03.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 >> 172.21.15.190:0/2007062097 conn(0x7f2a2c0f09a0 msgr2=0x7f2a2c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:03.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 shutdown_connections 2023-11-15T04:34:03.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:03.242+0000 7f2a2bfff700 1 -- 172.21.15.190:0/2007062097 wait complete. 2023-11-15T04:34:03.245 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:03.103515+0000 mgr.smithi047.zgysuz (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:03.241863+0000 mon.smithi047 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.190:0/2007062097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:04.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:04.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:05.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 -- 172.21.15.190:0/2466897449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 msgr2=0x7f3ac80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:05.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 --2- 172.21.15.190:0/2466897449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 0x7f3ac80f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f3ab8009fb0 tx=0x7f3ab8009b30).stop 2023-11-15T04:34:05.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 -- 172.21.15.190:0/2466897449 shutdown_connections 2023-11-15T04:34:05.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 --2- 172.21.15.190:0/2466897449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 0x7f3ac80f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:05.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 -- 172.21.15.190:0/2466897449 >> 172.21.15.190:0/2466897449 conn(0x7f3ac80f0b70 msgr2=0x7f3ac80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:05.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 -- 172.21.15.190:0/2466897449 shutdown_connections 2023-11-15T04:34:05.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 -- 172.21.15.190:0/2466897449 wait complete. 2023-11-15T04:34:05.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 Processor -- start 2023-11-15T04:34:05.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 -- start start 2023-11-15T04:34:05.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 0x7f3ac8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:05.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.730+0000 7f3ad053d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac8106c10 con 0x7f3ac80f5410 2023-11-15T04:34:05.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.731+0000 7f3ace2d9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 0x7f3ac8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:05.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.731+0000 7f3ace2d9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 0x7f3ac8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:44116/0 (socket says 172.21.15.190:44116) 2023-11-15T04:34:05.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.731+0000 7f3ace2d9700 1 -- 172.21.15.190:0/2614985102 learned_addr learned my addr 172.21.15.190:0/2614985102 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:05.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.731+0000 7f3ace2d9700 1 -- 172.21.15.190:0/2614985102 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab8005040 con 0x7f3ac80f5410 2023-11-15T04:34:05.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.731+0000 7f3ace2d9700 1 --2- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 0x7f3ac8106720 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f3ab8013510 tx=0x7f3ab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:05.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.731+0000 7f3abeffd700 1 -- 172.21.15.190:0/2614985102 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ab8009320 con 0x7f3ac80f5410 2023-11-15T04:34:05.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.732+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac8106e10 con 0x7f3ac80f5410 2023-11-15T04:34:05.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.732+0000 7f3abeffd700 1 -- 172.21.15.190:0/2614985102 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ab8013f00 con 0x7f3ac80f5410 2023-11-15T04:34:05.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.732+0000 7f3abeffd700 1 -- 172.21.15.190:0/2614985102 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ab8014d90 con 0x7f3ac80f5410 2023-11-15T04:34:05.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.732+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac81072a0 con 0x7f3ac80f5410 2023-11-15T04:34:05.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.732+0000 7f3abeffd700 1 -- 172.21.15.190:0/2614985102 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ab80263a0 con 0x7f3ac80f5410 2023-11-15T04:34:05.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.732+0000 7f3abeffd700 1 --2- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3ab402ff60 0x7f3ab4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:05.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.733+0000 7f3abeffd700 1 -- 172.21.15.190:0/2614985102 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ab8041300 con 0x7f3ac80f5410 2023-11-15T04:34:05.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.733+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac8045ad0 con 0x7f3ac80f5410 2023-11-15T04:34:05.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.733+0000 7f3acdad8700 1 --2- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3ab402ff60 0x7f3ab4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:05.738 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.733+0000 7f3acdad8700 1 --2- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3ab402ff60 0x7f3ab4032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3ac40096a0 tx=0x7f3ac4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:05.738 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.737+0000 7f3abeffd700 1 -- 172.21.15.190:0/2614985102 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ab8012920 con 0x7f3ac80f5410 2023-11-15T04:34:05.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.945+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ac80faf50 con 0x7f3ac80f5410 2023-11-15T04:34:05.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.945+0000 7f3abeffd700 1 -- 172.21.15.190:0/2614985102 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ab802b030 con 0x7f3ac80f5410 2023-11-15T04:34:05.946 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:05.947 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:05.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3ab402ff60 msgr2=0x7f3ab4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:05.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 --2- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3ab402ff60 0x7f3ab4032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3ac40096a0 tx=0x7f3ac4006b40).stop 2023-11-15T04:34:05.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 msgr2=0x7f3ac8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:05.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 --2- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 0x7f3ac8106720 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f3ab8013510 tx=0x7f3ab800b040).stop 2023-11-15T04:34:05.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 shutdown_connections 2023-11-15T04:34:05.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 --2- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3ab402ff60 0x7f3ab4032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:05.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 --2- 172.21.15.190:0/2614985102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ac80f5410 0x7f3ac8106720 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:05.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 >> 172.21.15.190:0/2614985102 conn(0x7f3ac80f0b70 msgr2=0x7f3ac805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:05.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 shutdown_connections 2023-11-15T04:34:05.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:05.947+0000 7f3ad053d700 1 -- 172.21.15.190:0/2614985102 wait complete. 2023-11-15T04:34:05.950 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:05.103849+0000 mgr.smithi047.zgysuz (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:05.218404+0000 mon.smithi047 (mon.0) 224 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:34:06.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:05.220421+0000 mon.smithi047 (mon.0) 225 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:34:06.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:05.946752+0000 mon.smithi047 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.190:0/2614985102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:07.377 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:07.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:08.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:07.104164+0000 mgr.smithi047.zgysuz (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:08.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.555+0000 7fd954aa2700 1 -- 172.21.15.190:0/1608344500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 msgr2=0x7fd9500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:08.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.555+0000 7fd954aa2700 1 --2- 172.21.15.190:0/1608344500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 0x7fd9500f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fd938009fb0 tx=0x7fd938009b30).stop 2023-11-15T04:34:08.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.555+0000 7fd954aa2700 1 -- 172.21.15.190:0/1608344500 shutdown_connections 2023-11-15T04:34:08.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.555+0000 7fd954aa2700 1 --2- 172.21.15.190:0/1608344500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 0x7fd9500f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:08.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.555+0000 7fd954aa2700 1 -- 172.21.15.190:0/1608344500 >> 172.21.15.190:0/1608344500 conn(0x7fd9500f0b70 msgr2=0x7fd9500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:08.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.555+0000 7fd954aa2700 1 -- 172.21.15.190:0/1608344500 shutdown_connections 2023-11-15T04:34:08.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.555+0000 7fd954aa2700 1 -- 172.21.15.190:0/1608344500 wait complete. 2023-11-15T04:34:08.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.556+0000 7fd954aa2700 1 Processor -- start 2023-11-15T04:34:08.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.556+0000 7fd954aa2700 1 -- start start 2023-11-15T04:34:08.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.556+0000 7fd954aa2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 0x7fd95010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:08.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.556+0000 7fd954aa2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd950107240 con 0x7fd9500f5410 2023-11-15T04:34:08.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.556+0000 7fd94e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 0x7fd95010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:08.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.556+0000 7fd94e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 0x7fd95010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:44138/0 (socket says 172.21.15.190:44138) 2023-11-15T04:34:08.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.556+0000 7fd94e59c700 1 -- 172.21.15.190:0/3365227992 learned_addr learned my addr 172.21.15.190:0/3365227992 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:08.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.557+0000 7fd94e59c700 1 -- 172.21.15.190:0/3365227992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd938005040 con 0x7fd9500f5410 2023-11-15T04:34:08.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.557+0000 7fd94e59c700 1 --2- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 0x7fd95010b850 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fd938014000 tx=0x7fd93800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:08.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.557+0000 7fd946ffd700 1 -- 172.21.15.190:0/3365227992 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9380095f0 con 0x7fd9500f5410 2023-11-15T04:34:08.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.557+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd950060220 con 0x7fd9500f5410 2023-11-15T04:34:08.562 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.557+0000 7fd946ffd700 1 -- 172.21.15.190:0/3365227992 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd938007e70 con 0x7fd9500f5410 2023-11-15T04:34:08.562 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.557+0000 7fd946ffd700 1 -- 172.21.15.190:0/3365227992 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd938014ea0 con 0x7fd9500f5410 2023-11-15T04:34:08.562 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.557+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95010bd40 con 0x7fd9500f5410 2023-11-15T04:34:08.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.558+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd950045ad0 con 0x7fd9500f5410 2023-11-15T04:34:08.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.558+0000 7fd946ffd700 1 -- 172.21.15.190:0/3365227992 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd938025070 con 0x7fd9500f5410 2023-11-15T04:34:08.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.558+0000 7fd946ffd700 1 --2- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd93c02ff60 0x7fd93c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:08.564 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.558+0000 7fd946ffd700 1 -- 172.21.15.190:0/3365227992 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd938040930 con 0x7fd9500f5410 2023-11-15T04:34:08.564 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.558+0000 7fd94dd9b700 1 --2- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd93c02ff60 0x7fd93c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:08.565 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.559+0000 7fd94dd9b700 1 --2- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd93c02ff60 0x7fd93c032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd9400096a0 tx=0x7fd940006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:08.566 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.563+0000 7fd946ffd700 1 -- 172.21.15.190:0/3365227992 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd93800f4e0 con 0x7fd9500f5410 2023-11-15T04:34:08.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.768+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9500fae60 con 0x7fd9500f5410 2023-11-15T04:34:08.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.768+0000 7fd946ffd700 1 -- 172.21.15.190:0/3365227992 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd93800f300 con 0x7fd9500f5410 2023-11-15T04:34:08.769 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:08.770 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:08.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd93c02ff60 msgr2=0x7fd93c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:08.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 --2- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd93c02ff60 0x7fd93c032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd9400096a0 tx=0x7fd940006b40).stop 2023-11-15T04:34:08.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 msgr2=0x7fd95010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:08.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 --2- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 0x7fd95010b850 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fd938014000 tx=0x7fd93800b040).stop 2023-11-15T04:34:08.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 shutdown_connections 2023-11-15T04:34:08.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 --2- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd93c02ff60 0x7fd93c032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:08.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 --2- 172.21.15.190:0/3365227992 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500f5410 0x7fd95010b850 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:08.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 >> 172.21.15.190:0/3365227992 conn(0x7fd9500f0b70 msgr2=0x7fd9500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:08.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 shutdown_connections 2023-11-15T04:34:08.773 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:08.770+0000 7fd954aa2700 1 -- 172.21.15.190:0/3365227992 wait complete. 2023-11-15T04:34:08.773 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:09.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:08.769924+0000 mon.smithi047 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.190:0/3365227992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:10.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:10.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:10.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:09.104512+0000 mgr.smithi047.zgysuz (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:11.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 -- 172.21.15.190:0/422230861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 msgr2=0x7fa4380f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:11.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 --2- 172.21.15.190:0/422230861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 0x7fa4380f66c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fa428009fb0 tx=0x7fa428009b30).stop 2023-11-15T04:34:11.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 -- 172.21.15.190:0/422230861 shutdown_connections 2023-11-15T04:34:11.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 --2- 172.21.15.190:0/422230861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 0x7fa4380f66c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 -- 172.21.15.190:0/422230861 >> 172.21.15.190:0/422230861 conn(0x7fa4380f1aa0 msgr2=0x7fa4380f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 -- 172.21.15.190:0/422230861 shutdown_connections 2023-11-15T04:34:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 -- 172.21.15.190:0/422230861 wait complete. 2023-11-15T04:34:11.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 Processor -- start 2023-11-15T04:34:11.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 -- start start 2023-11-15T04:34:11.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 0x7fa43806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:11.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.372+0000 7fa43ef30700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438069e20 con 0x7fa4380f62a0 2023-11-15T04:34:11.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa43df2e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 0x7fa43806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:11.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa43df2e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 0x7fa43806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34882/0 (socket says 172.21.15.190:34882) 2023-11-15T04:34:11.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa43df2e700 1 -- 172.21.15.190:0/2599610711 learned_addr learned my addr 172.21.15.190:0/2599610711 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:11.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa43df2e700 1 -- 172.21.15.190:0/2599610711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa428005040 con 0x7fa4380f62a0 2023-11-15T04:34:11.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa43df2e700 1 --2- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 0x7fa43806b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fa428013510 tx=0x7fa42800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:11.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa42e7fc700 1 -- 172.21.15.190:0/2599610711 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa428009320 con 0x7fa4380f62a0 2023-11-15T04:34:11.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa438069fc0 con 0x7fa4380f62a0 2023-11-15T04:34:11.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa42e7fc700 1 -- 172.21.15.190:0/2599610711 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa428013f00 con 0x7fa4380f62a0 2023-11-15T04:34:11.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa42e7fc700 1 -- 172.21.15.190:0/2599610711 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa428014dc0 con 0x7fa4380f62a0 2023-11-15T04:34:11.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.373+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43806a450 con 0x7fa4380f62a0 2023-11-15T04:34:11.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.374+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa438059100 con 0x7fa4380f62a0 2023-11-15T04:34:11.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.374+0000 7fa42e7fc700 1 -- 172.21.15.190:0/2599610711 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4280095f0 con 0x7fa4380f62a0 2023-11-15T04:34:11.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.374+0000 7fa42e7fc700 1 --2- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa42402ff60 0x7fa424032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:11.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.374+0000 7fa42e7fc700 1 -- 172.21.15.190:0/2599610711 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa428041000 con 0x7fa4380f62a0 2023-11-15T04:34:11.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.375+0000 7fa43d72d700 1 --2- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa42402ff60 0x7fa424032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:11.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.375+0000 7fa43d72d700 1 --2- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa42402ff60 0x7fa424032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa4340096a0 tx=0x7fa434006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:11.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.379+0000 7fa42e7fc700 1 -- 172.21.15.190:0/2599610711 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa428014420 con 0x7fa4380f62a0 2023-11-15T04:34:11.585 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.583+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4380fbce0 con 0x7fa4380f62a0 2023-11-15T04:34:11.585 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.584+0000 7fa42e7fc700 1 -- 172.21.15.190:0/2599610711 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa428010230 con 0x7fa4380f62a0 2023-11-15T04:34:11.586 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:11.586 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa42402ff60 msgr2=0x7fa424032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 --2- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa42402ff60 0x7fa424032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa4340096a0 tx=0x7fa434006b40).stop 2023-11-15T04:34:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 msgr2=0x7fa43806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 --2- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 0x7fa43806b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fa428013510 tx=0x7fa42800b040).stop 2023-11-15T04:34:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 shutdown_connections 2023-11-15T04:34:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 --2- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa42402ff60 0x7fa424032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 --2- 172.21.15.190:0/2599610711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4380f62a0 0x7fa43806b9a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 >> 172.21.15.190:0/2599610711 conn(0x7fa4380f1aa0 msgr2=0x7fa4380f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 shutdown_connections 2023-11-15T04:34:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:11.585+0000 7fa43ef30700 1 -- 172.21.15.190:0/2599610711 wait complete. 2023-11-15T04:34:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:12.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:12 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:11.104851+0000 mgr.smithi047.zgysuz (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:12.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:12 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:11.585465+0000 mon.smithi047 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.190:0/2599610711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:12.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:12.972 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:14.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.117+0000 7fc5a10ad700 1 -- 172.21.15.190:0/2174985963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 msgr2=0x7fc59c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:14.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.117+0000 7fc5a10ad700 1 --2- 172.21.15.190:0/2174985963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 0x7fc59c0f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fc590009fb0 tx=0x7fc590009b30).stop 2023-11-15T04:34:14.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.117+0000 7fc5a10ad700 1 -- 172.21.15.190:0/2174985963 shutdown_connections 2023-11-15T04:34:14.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.117+0000 7fc5a10ad700 1 --2- 172.21.15.190:0/2174985963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 0x7fc59c0f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:14.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.117+0000 7fc5a10ad700 1 -- 172.21.15.190:0/2174985963 >> 172.21.15.190:0/2174985963 conn(0x7fc59c0f1c70 msgr2=0x7fc59c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:14.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.117+0000 7fc5a10ad700 1 -- 172.21.15.190:0/2174985963 shutdown_connections 2023-11-15T04:34:14.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.117+0000 7fc5a10ad700 1 -- 172.21.15.190:0/2174985963 wait complete. 2023-11-15T04:34:14.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.117+0000 7fc5a10ad700 1 Processor -- start 2023-11-15T04:34:14.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.117+0000 7fc5a10ad700 1 -- start start 2023-11-15T04:34:14.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.118+0000 7fc5a10ad700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 0x7fc59c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:14.121 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.118+0000 7fc5a10ad700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c106c20 con 0x7fc59c0f6510 2023-11-15T04:34:14.122 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.118+0000 7fc59ad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 0x7fc59c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:14.122 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.118+0000 7fc59ad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 0x7fc59c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34902/0 (socket says 172.21.15.190:34902) 2023-11-15T04:34:14.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.118+0000 7fc59ad9d700 1 -- 172.21.15.190:0/1784728514 learned_addr learned my addr 172.21.15.190:0/1784728514 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:14.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.118+0000 7fc59ad9d700 1 -- 172.21.15.190:0/1784728514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc590005040 con 0x7fc59c0f6510 2023-11-15T04:34:14.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.118+0000 7fc59ad9d700 1 --2- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 0x7fc59c106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc590013510 tx=0x7fc59000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:14.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.119+0000 7fc5837fe700 1 -- 172.21.15.190:0/1784728514 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc590009320 con 0x7fc59c0f6510 2023-11-15T04:34:14.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.119+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c106e20 con 0x7fc59c0f6510 2023-11-15T04:34:14.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.119+0000 7fc5837fe700 1 -- 172.21.15.190:0/1784728514 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc590013f00 con 0x7fc59c0f6510 2023-11-15T04:34:14.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.119+0000 7fc5837fe700 1 -- 172.21.15.190:0/1784728514 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc590014d90 con 0x7fc59c0f6510 2023-11-15T04:34:14.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.119+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c1072b0 con 0x7fc59c0f6510 2023-11-15T04:34:14.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.120+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c045ad0 con 0x7fc59c0f6510 2023-11-15T04:34:14.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.120+0000 7fc5837fe700 1 -- 172.21.15.190:0/1784728514 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5900095f0 con 0x7fc59c0f6510 2023-11-15T04:34:14.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.120+0000 7fc5837fe700 1 --2- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc58402ff60 0x7fc584032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:14.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.120+0000 7fc5837fe700 1 -- 172.21.15.190:0/1784728514 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc590040fe0 con 0x7fc59c0f6510 2023-11-15T04:34:14.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.120+0000 7fc59a59c700 1 --2- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc58402ff60 0x7fc584032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:14.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.120+0000 7fc59a59c700 1 --2- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc58402ff60 0x7fc584032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc58c0096a0 tx=0x7fc58c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:14.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.125+0000 7fc5837fe700 1 -- 172.21.15.190:0/1784728514 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc59001d7f0 con 0x7fc59c0f6510 2023-11-15T04:34:14.336 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.335+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc59c0fbf60 con 0x7fc59c0f6510 2023-11-15T04:34:14.337 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.336+0000 7fc5837fe700 1 -- 172.21.15.190:0/1784728514 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc590010230 con 0x7fc59c0f6510 2023-11-15T04:34:14.337 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:14.338 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:14.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.337+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc58402ff60 msgr2=0x7fc584032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:14.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.337+0000 7fc5a10ad700 1 --2- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc58402ff60 0x7fc584032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc58c0096a0 tx=0x7fc58c006b40).stop 2023-11-15T04:34:14.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.337+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 msgr2=0x7fc59c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:14.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.337+0000 7fc5a10ad700 1 --2- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 0x7fc59c106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc590013510 tx=0x7fc59000b040).stop 2023-11-15T04:34:14.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.337+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 shutdown_connections 2023-11-15T04:34:14.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.337+0000 7fc5a10ad700 1 --2- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc58402ff60 0x7fc584032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:14.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.337+0000 7fc5a10ad700 1 --2- 172.21.15.190:0/1784728514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc59c0f6510 0x7fc59c106730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:14.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.337+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 >> 172.21.15.190:0/1784728514 conn(0x7fc59c0f1c70 msgr2=0x7fc59c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:14.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.338+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 shutdown_connections 2023-11-15T04:34:14.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:14.338+0000 7fc5a10ad700 1 -- 172.21.15.190:0/1784728514 wait complete. 2023-11-15T04:34:14.341 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:13.105160+0000 mgr.smithi047.zgysuz (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:15.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:15 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:14.337401+0000 mon.smithi047 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.190:0/1784728514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:15.727 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:15.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:15.105497+0000 mgr.smithi047.zgysuz (mgr.14164) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:16.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.845+0000 7f333b98c700 1 -- 172.21.15.190:0/4170168842 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 msgr2=0x7f33340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:16.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.845+0000 7f333b98c700 1 --2- 172.21.15.190:0/4170168842 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 0x7f33340f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f3330009fb0 tx=0x7f3330009b30).stop 2023-11-15T04:34:16.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.845+0000 7f333b98c700 1 -- 172.21.15.190:0/4170168842 shutdown_connections 2023-11-15T04:34:16.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.845+0000 7f333b98c700 1 --2- 172.21.15.190:0/4170168842 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 0x7f33340f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:16.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.845+0000 7f333b98c700 1 -- 172.21.15.190:0/4170168842 >> 172.21.15.190:0/4170168842 conn(0x7f33340f0b70 msgr2=0x7f33340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:16.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.845+0000 7f333b98c700 1 -- 172.21.15.190:0/4170168842 shutdown_connections 2023-11-15T04:34:16.848 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.845+0000 7f333b98c700 1 -- 172.21.15.190:0/4170168842 wait complete. 2023-11-15T04:34:16.849 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.846+0000 7f333b98c700 1 Processor -- start 2023-11-15T04:34:16.849 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.846+0000 7f333b98c700 1 -- start start 2023-11-15T04:34:16.849 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.846+0000 7f333b98c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 0x7f3334105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:16.849 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.846+0000 7f333b98c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3334105b20 con 0x7f33340f5410 2023-11-15T04:34:16.849 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.846+0000 7f3339728700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 0x7f3334105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:16.850 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.846+0000 7f3339728700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 0x7f3334105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34916/0 (socket says 172.21.15.190:34916) 2023-11-15T04:34:16.850 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.846+0000 7f3339728700 1 -- 172.21.15.190:0/3054271585 learned_addr learned my addr 172.21.15.190:0/3054271585 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:16.850 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.847+0000 7f3339728700 1 -- 172.21.15.190:0/3054271585 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330005040 con 0x7f33340f5410 2023-11-15T04:34:16.851 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.847+0000 7f3339728700 1 --2- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 0x7f3334105630 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f3330013510 tx=0x7f333000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:16.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.847+0000 7f3329ffb700 1 -- 172.21.15.190:0/3054271585 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3330009320 con 0x7f33340f5410 2023-11-15T04:34:16.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.847+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3334105d20 con 0x7f33340f5410 2023-11-15T04:34:16.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.847+0000 7f3329ffb700 1 -- 172.21.15.190:0/3054271585 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3330013f00 con 0x7f33340f5410 2023-11-15T04:34:16.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.847+0000 7f3329ffb700 1 -- 172.21.15.190:0/3054271585 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3330014d90 con 0x7f33340f5410 2023-11-15T04:34:16.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.847+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33341061b0 con 0x7f33340f5410 2023-11-15T04:34:16.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.848+0000 7f3329ffb700 1 -- 172.21.15.190:0/3054271585 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3330014420 con 0x7f33340f5410 2023-11-15T04:34:16.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.848+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3334045ad0 con 0x7f33340f5410 2023-11-15T04:34:16.854 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.848+0000 7f3329ffb700 1 --2- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f332002ffb0 0x7f3320032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:16.854 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.848+0000 7f3329ffb700 1 -- 172.21.15.190:0/3054271585 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33300410b0 con 0x7f33340f5410 2023-11-15T04:34:16.854 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.848+0000 7f3338f27700 1 --2- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f332002ffb0 0x7f3320032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:16.854 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.849+0000 7f3338f27700 1 --2- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f332002ffb0 0x7f3320032470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f33240096a0 tx=0x7f3324006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:16.855 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:16.853+0000 7f3329ffb700 1 -- 172.21.15.190:0/3054271585 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f333006d2b0 con 0x7f33340f5410 2023-11-15T04:34:17.060 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.059+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33340fae60 con 0x7f33340f5410 2023-11-15T04:34:17.060 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.060+0000 7f3329ffb700 1 -- 172.21.15.190:0/3054271585 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f33300095f0 con 0x7f33340f5410 2023-11-15T04:34:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:17.061 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:17.062 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.061+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f332002ffb0 msgr2=0x7f3320032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:17.063 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.061+0000 7f333b98c700 1 --2- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f332002ffb0 0x7f3320032470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f33240096a0 tx=0x7f3324006b40).stop 2023-11-15T04:34:17.063 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.062+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 msgr2=0x7f3334105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:17.063 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.062+0000 7f333b98c700 1 --2- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 0x7f3334105630 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f3330013510 tx=0x7f333000b040).stop 2023-11-15T04:34:17.063 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.062+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 shutdown_connections 2023-11-15T04:34:17.063 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.062+0000 7f333b98c700 1 --2- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f332002ffb0 0x7f3320032470 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:17.064 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.062+0000 7f333b98c700 1 --2- 172.21.15.190:0/3054271585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33340f5410 0x7f3334105630 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:17.064 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.062+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 >> 172.21.15.190:0/3054271585 conn(0x7f33340f0b70 msgr2=0x7f33340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:17.064 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.062+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 shutdown_connections 2023-11-15T04:34:17.064 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:17.062+0000 7f333b98c700 1 -- 172.21.15.190:0/3054271585 wait complete. 2023-11-15T04:34:17.065 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:17.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:34:16.328491+0000 mgr.smithi047.zgysuz (mgr.14164) 157 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi047 ... 2023-11-15T04:34:17.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T04:34:17.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T04:34:17.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:29:53Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:30:30Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:31:08Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T04:34:17.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T04:34:17.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T04:34:17.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:32:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:32:59Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:33:36Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.261 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T04:34:17.261 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Traceback (most recent call last): 2023-11-15T04:34:17.261 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T04:34:17.261 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: main() 2023-11-15T04:34:17.261 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T04:34:17.262 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: r = ctx.func(ctx) 2023-11-15T04:34:17.262 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T04:34:17.262 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: return func(ctx) 2023-11-15T04:34:17.262 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T04:34:17.262 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T04:34:17.263 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T04:34:17.263 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T04:34:17.263 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T04:34:17.263 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: raise RuntimeError('uid/gid not found') 2023-11-15T04:34:17.263 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: RuntimeError: uid/gid not found 2023-11-15T04:34:17.264 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Traceback (most recent call last): 2023-11-15T04:34:17.264 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T04:34:17.264 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: yield (conn, connr) 2023-11-15T04:34:17.264 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T04:34:17.264 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: code, '\n'.join(err))) 2023-11-15T04:34:17.265 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi047 ... 2023-11-15T04:34:17.265 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T04:34:17.265 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T04:34:17.265 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:29:53Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.266 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:30:30Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.266 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:31:08Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.266 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T04:34:17.266 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T04:34:17.266 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T04:34:17.267 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:32:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.267 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:32:59Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.267 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr time="2023-11-15T04:33:36Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:34:17.267 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T04:34:17.267 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Traceback (most recent call last): 2023-11-15T04:34:17.268 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T04:34:17.268 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: main() 2023-11-15T04:34:17.268 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T04:34:17.268 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: r = ctx.func(ctx) 2023-11-15T04:34:17.268 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T04:34:17.269 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: return func(ctx) 2023-11-15T04:34:17.269 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T04:34:17.269 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T04:34:17.269 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T04:34:17.269 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T04:34:17.270 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T04:34:17.270 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: raise RuntimeError('uid/gid not found') 2023-11-15T04:34:17.270 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: RuntimeError: uid/gid not found 2023-11-15T04:34:17.270 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:16.329818+0000 mon.smithi047 (mon.0) 230 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-15T04:34:17.270 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:16.332910+0000 mon.smithi047 (mon.0) 231 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-15T04:34:17.271 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:16.333829+0000 mon.smithi047 (mon.0) 232 : audit [DBG] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T04:34:17.271 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:34:16.334593+0000 mgr.smithi047.zgysuz (mgr.14164) 158 : cephadm [INF] Deploying daemon crash.smithi047 on smithi047 2023-11-15T04:34:17.271 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:17.061219+0000 mon.smithi047 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.190:0/3054271585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:18.163 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:17.105800+0000 mgr.smithi047.zgysuz (mgr.14164) 159 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:18.164 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:17.819059+0000 mon.smithi047 (mon.0) 234 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:34:18.164 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:17.963109+0000 mon.smithi047 (mon.0) 235 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:34:18.164 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:17.965492+0000 mon.smithi047 (mon.0) 236 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:34:18.164 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:17.966049+0000 mon.smithi047 (mon.0) 237 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T04:34:18.165 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:17.969096+0000 mon.smithi047 (mon.0) 238 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:34:18.442 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:18.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:19.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:19 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:17.966528+0000 mgr.smithi047.zgysuz (mgr.14164) 160 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T04:34:19.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:19 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:34:17.972608+0000 mgr.smithi047.zgysuz (mgr.14164) 161 : cephadm [INF] Deploying daemon grafana.smithi047 on smithi047 2023-11-15T04:34:19.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.591+0000 7ff1f09a0700 1 -- 172.21.15.190:0/1248210547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 msgr2=0x7ff1ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:19.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.591+0000 7ff1f09a0700 1 --2- 172.21.15.190:0/1248210547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 0x7ff1ec0f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7ff1dc009fb0 tx=0x7ff1dc009b30).stop 2023-11-15T04:34:19.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.591+0000 7ff1f09a0700 1 -- 172.21.15.190:0/1248210547 shutdown_connections 2023-11-15T04:34:19.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.591+0000 7ff1f09a0700 1 --2- 172.21.15.190:0/1248210547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 0x7ff1ec0f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:19.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.591+0000 7ff1f09a0700 1 -- 172.21.15.190:0/1248210547 >> 172.21.15.190:0/1248210547 conn(0x7ff1ec0f0b70 msgr2=0x7ff1ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:19.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.591+0000 7ff1f09a0700 1 -- 172.21.15.190:0/1248210547 shutdown_connections 2023-11-15T04:34:19.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.591+0000 7ff1f09a0700 1 -- 172.21.15.190:0/1248210547 wait complete. 2023-11-15T04:34:19.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.592+0000 7ff1f09a0700 1 Processor -- start 2023-11-15T04:34:19.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.592+0000 7ff1f09a0700 1 -- start start 2023-11-15T04:34:19.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.592+0000 7ff1f09a0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 0x7ff1ec109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:19.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.592+0000 7ff1f09a0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec060220 con 0x7ff1ec0f5410 2023-11-15T04:34:19.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.592+0000 7ff1ea59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 0x7ff1ec109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:19.596 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.592+0000 7ff1ea59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 0x7ff1ec109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34938/0 (socket says 172.21.15.190:34938) 2023-11-15T04:34:19.596 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.592+0000 7ff1ea59c700 1 -- 172.21.15.190:0/2242830991 learned_addr learned my addr 172.21.15.190:0/2242830991 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:19.597 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.593+0000 7ff1ea59c700 1 -- 172.21.15.190:0/2242830991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc005040 con 0x7ff1ec0f5410 2023-11-15T04:34:19.597 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.593+0000 7ff1ea59c700 1 --2- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 0x7ff1ec109450 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7ff1dc014000 tx=0x7ff1dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.593+0000 7ff1e2ffd700 1 -- 172.21.15.190:0/2242830991 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1dc0095f0 con 0x7ff1ec0f5410 2023-11-15T04:34:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.593+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ec060420 con 0x7ff1ec0f5410 2023-11-15T04:34:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.593+0000 7ff1e2ffd700 1 -- 172.21.15.190:0/2242830991 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1dc007e70 con 0x7ff1ec0f5410 2023-11-15T04:34:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.593+0000 7ff1e2ffd700 1 -- 172.21.15.190:0/2242830991 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1dc014ea0 con 0x7ff1ec0f5410 2023-11-15T04:34:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.593+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ec109b00 con 0x7ff1ec0f5410 2023-11-15T04:34:19.599 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.594+0000 7ff1e2ffd700 1 -- 172.21.15.190:0/2242830991 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1dc025070 con 0x7ff1ec0f5410 2023-11-15T04:34:19.599 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.594+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ec059100 con 0x7ff1ec0f5410 2023-11-15T04:34:19.599 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.594+0000 7ff1e2ffd700 1 --2- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1d802ff60 0x7ff1d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:19.599 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.594+0000 7ff1e2ffd700 1 -- 172.21.15.190:0/2242830991 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1dc040970 con 0x7ff1ec0f5410 2023-11-15T04:34:19.600 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.594+0000 7ff1e9d9b700 1 --2- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1d802ff60 0x7ff1d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:19.600 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.595+0000 7ff1e9d9b700 1 --2- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1d802ff60 0x7ff1d8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff1d40096a0 tx=0x7ff1d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:19.600 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.599+0000 7ff1e2ffd700 1 -- 172.21.15.190:0/2242830991 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1dc0079c0 con 0x7ff1ec0f5410 2023-11-15T04:34:19.808 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.807+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1ec045ad0 con 0x7ff1ec0f5410 2023-11-15T04:34:19.809 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.808+0000 7ff1e2ffd700 1 -- 172.21.15.190:0/2242830991 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff1dc00f300 con 0x7ff1ec0f5410 2023-11-15T04:34:19.809 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:19.809 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:19.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1d802ff60 msgr2=0x7ff1d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:19.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 --2- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1d802ff60 0x7ff1d8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff1d40096a0 tx=0x7ff1d4006b40).stop 2023-11-15T04:34:19.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 msgr2=0x7ff1ec109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:19.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 --2- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 0x7ff1ec109450 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7ff1dc014000 tx=0x7ff1dc00b040).stop 2023-11-15T04:34:19.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 shutdown_connections 2023-11-15T04:34:19.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 --2- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff1d802ff60 0x7ff1d8032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:19.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 --2- 172.21.15.190:0/2242830991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ec0f5410 0x7ff1ec109450 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:19.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 >> 172.21.15.190:0/2242830991 conn(0x7ff1ec0f0b70 msgr2=0x7ff1ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:19.812 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 shutdown_connections 2023-11-15T04:34:19.812 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:19.809+0000 7ff1f09a0700 1 -- 172.21.15.190:0/2242830991 wait complete. 2023-11-15T04:34:19.812 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:20.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:20 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:19.106177+0000 mgr.smithi047.zgysuz (mgr.14164) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:20.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:20 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:19.809328+0000 mon.smithi047 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.190:0/2242830991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:21.196 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:21.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:22.320 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.317+0000 7f7fc3195700 1 -- 172.21.15.190:0/3429862287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 msgr2=0x7f7fbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:22.320 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.317+0000 7f7fc3195700 1 --2- 172.21.15.190:0/3429862287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 0x7f7fbc0f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f7fb0009fb0 tx=0x7f7fb0009b30).stop 2023-11-15T04:34:22.320 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.317+0000 7f7fc3195700 1 -- 172.21.15.190:0/3429862287 shutdown_connections 2023-11-15T04:34:22.320 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.317+0000 7f7fc3195700 1 --2- 172.21.15.190:0/3429862287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 0x7f7fbc0f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:22.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.317+0000 7f7fc3195700 1 -- 172.21.15.190:0/3429862287 >> 172.21.15.190:0/3429862287 conn(0x7f7fbc0f0b70 msgr2=0x7f7fbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:22.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.317+0000 7f7fc3195700 1 -- 172.21.15.190:0/3429862287 shutdown_connections 2023-11-15T04:34:22.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.317+0000 7f7fc3195700 1 -- 172.21.15.190:0/3429862287 wait complete. 2023-11-15T04:34:22.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.318+0000 7f7fc3195700 1 Processor -- start 2023-11-15T04:34:22.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.318+0000 7f7fc3195700 1 -- start start 2023-11-15T04:34:22.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.318+0000 7f7fc3195700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 0x7f7fbc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:22.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.318+0000 7f7fc3195700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc105b20 con 0x7f7fbc0f5410 2023-11-15T04:34:22.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.318+0000 7f7fc0f31700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 0x7f7fbc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:22.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.318+0000 7f7fc0f31700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 0x7f7fbc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:57386/0 (socket says 172.21.15.190:57386) 2023-11-15T04:34:22.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.318+0000 7f7fc0f31700 1 -- 172.21.15.190:0/4026003602 learned_addr learned my addr 172.21.15.190:0/4026003602 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:22.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.318+0000 7f7fc0f31700 1 -- 172.21.15.190:0/4026003602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb0005040 con 0x7f7fbc0f5410 2023-11-15T04:34:22.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.319+0000 7f7fc0f31700 1 --2- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 0x7f7fbc105630 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f7fb0013510 tx=0x7f7fb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:22.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.319+0000 7f7fb97fa700 1 -- 172.21.15.190:0/4026003602 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fb0009320 con 0x7f7fbc0f5410 2023-11-15T04:34:22.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.319+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fbc105d20 con 0x7f7fbc0f5410 2023-11-15T04:34:22.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.319+0000 7f7fb97fa700 1 -- 172.21.15.190:0/4026003602 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fb0013f00 con 0x7f7fbc0f5410 2023-11-15T04:34:22.324 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.319+0000 7f7fb97fa700 1 -- 172.21.15.190:0/4026003602 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fb0014d90 con 0x7f7fbc0f5410 2023-11-15T04:34:22.324 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.319+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fbc1061b0 con 0x7f7fbc0f5410 2023-11-15T04:34:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.320+0000 7f7fb97fa700 1 -- 172.21.15.190:0/4026003602 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fb00095f0 con 0x7f7fbc0f5410 2023-11-15T04:34:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.320+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fbc045ad0 con 0x7f7fbc0f5410 2023-11-15T04:34:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.320+0000 7f7fb97fa700 1 --2- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7fac02ff60 0x7f7fac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.320+0000 7f7fb97fa700 1 -- 172.21.15.190:0/4026003602 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fb0041000 con 0x7f7fbc0f5410 2023-11-15T04:34:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.320+0000 7f7fbbfff700 1 --2- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7fac02ff60 0x7f7fac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.321+0000 7f7fbbfff700 1 --2- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7fac02ff60 0x7f7fac032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f7fa80096a0 tx=0x7f7fa8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.325+0000 7f7fb97fa700 1 -- 172.21.15.190:0/4026003602 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fb0014420 con 0x7f7fbc0f5410 2023-11-15T04:34:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:21.106510+0000 mgr.smithi047.zgysuz (mgr.14164) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:22.534 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.534+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fbc0fae60 con 0x7f7fbc0f5410 2023-11-15T04:34:22.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.534+0000 7f7fb97fa700 1 -- 172.21.15.190:0/4026003602 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7fb0010230 con 0x7f7fbc0f5410 2023-11-15T04:34:22.535 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:22.536 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:22.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7fac02ff60 msgr2=0x7f7fac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:22.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 --2- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7fac02ff60 0x7f7fac032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f7fa80096a0 tx=0x7f7fa8006b40).stop 2023-11-15T04:34:22.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 msgr2=0x7f7fbc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:22.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 --2- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 0x7f7fbc105630 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f7fb0013510 tx=0x7f7fb000b040).stop 2023-11-15T04:34:22.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 shutdown_connections 2023-11-15T04:34:22.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 --2- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7fac02ff60 0x7f7fac032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:22.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 --2- 172.21.15.190:0/4026003602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fbc0f5410 0x7f7fbc105630 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:22.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 >> 172.21.15.190:0/4026003602 conn(0x7f7fbc0f0b70 msgr2=0x7f7fbc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:22.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 shutdown_connections 2023-11-15T04:34:22.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:22.536+0000 7f7fc3195700 1 -- 172.21.15.190:0/4026003602 wait complete. 2023-11-15T04:34:22.539 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:23.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:23 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:22.535896+0000 mon.smithi047 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.190:0/4026003602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:24.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:24.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:24.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:23.106864+0000 mgr.smithi047.zgysuz (mgr.14164) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:25.141 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.140+0000 7f07d9813700 1 -- 172.21.15.190:0/2828512684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 msgr2=0x7f07d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:25.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.140+0000 7f07d9813700 1 --2- 172.21.15.190:0/2828512684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 0x7f07d40f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f07c4009fb0 tx=0x7f07c4009b30).stop 2023-11-15T04:34:25.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.140+0000 7f07d9813700 1 -- 172.21.15.190:0/2828512684 shutdown_connections 2023-11-15T04:34:25.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.140+0000 7f07d9813700 1 --2- 172.21.15.190:0/2828512684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 0x7f07d40f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:25.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.140+0000 7f07d9813700 1 -- 172.21.15.190:0/2828512684 >> 172.21.15.190:0/2828512684 conn(0x7f07d40f1c70 msgr2=0x7f07d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:25.143 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.140+0000 7f07d9813700 1 -- 172.21.15.190:0/2828512684 shutdown_connections 2023-11-15T04:34:25.143 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.140+0000 7f07d9813700 1 -- 172.21.15.190:0/2828512684 wait complete. 2023-11-15T04:34:25.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.141+0000 7f07d9813700 1 Processor -- start 2023-11-15T04:34:25.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.141+0000 7f07d9813700 1 -- start start 2023-11-15T04:34:25.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.141+0000 7f07d9813700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 0x7f07d4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:25.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.141+0000 7f07d9813700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4106c20 con 0x7f07d40f6510 2023-11-15T04:34:25.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.141+0000 7f07d2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 0x7f07d4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:25.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.141+0000 7f07d2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 0x7f07d4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:57404/0 (socket says 172.21.15.190:57404) 2023-11-15T04:34:25.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.141+0000 7f07d2ffd700 1 -- 172.21.15.190:0/1272337625 learned_addr learned my addr 172.21.15.190:0/1272337625 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:25.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.142+0000 7f07d2ffd700 1 -- 172.21.15.190:0/1272337625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4005040 con 0x7f07d40f6510 2023-11-15T04:34:25.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.142+0000 7f07d2ffd700 1 --2- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 0x7f07d4106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f07c4013510 tx=0x7f07c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:25.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.142+0000 7f07cb7fe700 1 -- 172.21.15.190:0/1272337625 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c4009320 con 0x7f07d40f6510 2023-11-15T04:34:25.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.142+0000 7f07cb7fe700 1 -- 172.21.15.190:0/1272337625 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07c4013f00 con 0x7f07d40f6510 2023-11-15T04:34:25.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.142+0000 7f07cb7fe700 1 -- 172.21.15.190:0/1272337625 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c4014d90 con 0x7f07d40f6510 2023-11-15T04:34:25.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.142+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d4106e20 con 0x7f07d40f6510 2023-11-15T04:34:25.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.142+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d41072b0 con 0x7f07d40f6510 2023-11-15T04:34:25.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.143+0000 7f07cb7fe700 1 -- 172.21.15.190:0/1272337625 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07c40095f0 con 0x7f07d40f6510 2023-11-15T04:34:25.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.143+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d4045ad0 con 0x7f07d40f6510 2023-11-15T04:34:25.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.143+0000 7f07cb7fe700 1 --2- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ffb0 0x7f07c0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:25.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.143+0000 7f07cb7fe700 1 -- 172.21.15.190:0/1272337625 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07c4040ff0 con 0x7f07d40f6510 2023-11-15T04:34:25.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.143+0000 7f07d27fc700 1 --2- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ffb0 0x7f07c0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:25.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.144+0000 7f07d27fc700 1 --2- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ffb0 0x7f07c0032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f07bc0096a0 tx=0x7f07bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:25.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.148+0000 7f07cb7fe700 1 -- 172.21.15.190:0/1272337625 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07c406d2b0 con 0x7f07d40f6510 2023-11-15T04:34:25.354 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.353+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07d40fbf60 con 0x7f07d40f6510 2023-11-15T04:34:25.355 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.354+0000 7f07cb7fe700 1 -- 172.21.15.190:0/1272337625 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07c4014420 con 0x7f07d40f6510 2023-11-15T04:34:25.355 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:25.356 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:25.356 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ffb0 msgr2=0x7f07c0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:25.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 --2- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ffb0 0x7f07c0032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f07bc0096a0 tx=0x7f07bc006b40).stop 2023-11-15T04:34:25.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 msgr2=0x7f07d4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:25.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 --2- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 0x7f07d4106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f07c4013510 tx=0x7f07c400b040).stop 2023-11-15T04:34:25.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 shutdown_connections 2023-11-15T04:34:25.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 --2- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f07c002ffb0 0x7f07c0032470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:25.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 --2- 172.21.15.190:0/1272337625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07d40f6510 0x7f07d4106730 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:25.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 >> 172.21.15.190:0/1272337625 conn(0x7f07d40f1c70 msgr2=0x7f07d40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:25.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 shutdown_connections 2023-11-15T04:34:25.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:25.356+0000 7f07d9813700 1 -- 172.21.15.190:0/1272337625 wait complete. 2023-11-15T04:34:25.359 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:26.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:25.107145+0000 mgr.smithi047.zgysuz (mgr.14164) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:26.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:25.355453+0000 mon.smithi047 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.190:0/1272337625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:26.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:26.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:27.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.897+0000 7f1c56cd9700 1 -- 172.21.15.190:0/3617999389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 msgr2=0x7f1c500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:27.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.897+0000 7f1c56cd9700 1 --2- 172.21.15.190:0/3617999389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 0x7f1c500f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f1c44009fb0 tx=0x7f1c44009b30).stop 2023-11-15T04:34:27.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.897+0000 7f1c56cd9700 1 -- 172.21.15.190:0/3617999389 shutdown_connections 2023-11-15T04:34:27.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.897+0000 7f1c56cd9700 1 --2- 172.21.15.190:0/3617999389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 0x7f1c500f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:27.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.897+0000 7f1c56cd9700 1 -- 172.21.15.190:0/3617999389 >> 172.21.15.190:0/3617999389 conn(0x7f1c500f0b70 msgr2=0x7f1c500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:27.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.897+0000 7f1c56cd9700 1 -- 172.21.15.190:0/3617999389 shutdown_connections 2023-11-15T04:34:27.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.897+0000 7f1c56cd9700 1 -- 172.21.15.190:0/3617999389 wait complete. 2023-11-15T04:34:27.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.898+0000 7f1c56cd9700 1 Processor -- start 2023-11-15T04:34:27.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.898+0000 7f1c56cd9700 1 -- start start 2023-11-15T04:34:27.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.898+0000 7f1c56cd9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 0x7f1c50109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:27.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.898+0000 7f1c56cd9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50060240 con 0x7f1c500f5410 2023-11-15T04:34:27.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.898+0000 7f1c54a75700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 0x7f1c50109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:27.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.898+0000 7f1c54a75700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 0x7f1c50109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:57426/0 (socket says 172.21.15.190:57426) 2023-11-15T04:34:27.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.898+0000 7f1c54a75700 1 -- 172.21.15.190:0/2803276261 learned_addr learned my addr 172.21.15.190:0/2803276261 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:27.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.898+0000 7f1c54a75700 1 -- 172.21.15.190:0/2803276261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c44005040 con 0x7f1c500f5410 2023-11-15T04:34:27.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.899+0000 7f1c54a75700 1 --2- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 0x7f1c50109430 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f1c44013510 tx=0x7f1c4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:27.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.899+0000 7f1c4d7fa700 1 -- 172.21.15.190:0/2803276261 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c44009320 con 0x7f1c500f5410 2023-11-15T04:34:27.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.899+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c50060440 con 0x7f1c500f5410 2023-11-15T04:34:27.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.899+0000 7f1c4d7fa700 1 -- 172.21.15.190:0/2803276261 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c44013f00 con 0x7f1c500f5410 2023-11-15T04:34:27.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.899+0000 7f1c4d7fa700 1 -- 172.21.15.190:0/2803276261 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c44014d90 con 0x7f1c500f5410 2023-11-15T04:34:27.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.899+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c50109ae0 con 0x7f1c500f5410 2023-11-15T04:34:27.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.900+0000 7f1c4d7fa700 1 -- 172.21.15.190:0/2803276261 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c44014420 con 0x7f1c500f5410 2023-11-15T04:34:27.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.900+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c50045ad0 con 0x7f1c500f5410 2023-11-15T04:34:27.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.900+0000 7f1c4d7fa700 1 --2- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1c4002ff60 0x7f1c40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:27.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.900+0000 7f1c4d7fa700 1 -- 172.21.15.190:0/2803276261 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c44041000 con 0x7f1c500f5410 2023-11-15T04:34:27.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.900+0000 7f1c4ffff700 1 --2- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1c4002ff60 0x7f1c40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:27.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.900+0000 7f1c4ffff700 1 --2- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1c4002ff60 0x7f1c40032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1c3c0096a0 tx=0x7f1c3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:27.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:27.905+0000 7f1c4d7fa700 1 -- 172.21.15.190:0/2803276261 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c440095f0 con 0x7f1c500f5410 2023-11-15T04:34:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:27.107380+0000 mgr.smithi047.zgysuz (mgr.14164) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:28.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.112+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c500fae60 con 0x7f1c500f5410 2023-11-15T04:34:28.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.112+0000 7f1c4d7fa700 1 -- 172.21.15.190:0/2803276261 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c4401db70 con 0x7f1c500f5410 2023-11-15T04:34:28.114 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:28.114 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:28.117 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1c4002ff60 msgr2=0x7f1c40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:28.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 --2- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1c4002ff60 0x7f1c40032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1c3c0096a0 tx=0x7f1c3c006b40).stop 2023-11-15T04:34:28.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 msgr2=0x7f1c50109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:28.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 --2- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 0x7f1c50109430 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f1c44013510 tx=0x7f1c4400b040).stop 2023-11-15T04:34:28.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 shutdown_connections 2023-11-15T04:34:28.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 --2- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1c4002ff60 0x7f1c40032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:28.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 --2- 172.21.15.190:0/2803276261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c500f5410 0x7f1c50109430 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:28.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 >> 172.21.15.190:0/2803276261 conn(0x7f1c500f0b70 msgr2=0x7f1c500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:28.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 shutdown_connections 2023-11-15T04:34:28.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:28.114+0000 7f1c56cd9700 1 -- 172.21.15.190:0/2803276261 wait complete. 2023-11-15T04:34:28.120 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:29.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:28 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:28.114094+0000 mon.smithi047 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.190:0/2803276261' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:29.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:29.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:30.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:29 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:29.107612+0000 mgr.smithi047.zgysuz (mgr.14164) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:30.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.634+0000 7f8fcca68700 1 -- 172.21.15.190:0/181926136 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 msgr2=0x7f8fc80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:30.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.634+0000 7f8fcca68700 1 --2- 172.21.15.190:0/181926136 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 0x7f8fc80f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f8fb8009fb0 tx=0x7f8fb8009b30).stop 2023-11-15T04:34:30.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.634+0000 7f8fcca68700 1 -- 172.21.15.190:0/181926136 shutdown_connections 2023-11-15T04:34:30.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.634+0000 7f8fcca68700 1 --2- 172.21.15.190:0/181926136 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 0x7f8fc80f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:30.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.634+0000 7f8fcca68700 1 -- 172.21.15.190:0/181926136 >> 172.21.15.190:0/181926136 conn(0x7f8fc80f0b70 msgr2=0x7f8fc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:30.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.634+0000 7f8fcca68700 1 -- 172.21.15.190:0/181926136 shutdown_connections 2023-11-15T04:34:30.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.634+0000 7f8fcca68700 1 -- 172.21.15.190:0/181926136 wait complete. 2023-11-15T04:34:30.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.635+0000 7f8fcca68700 1 Processor -- start 2023-11-15T04:34:30.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.635+0000 7f8fcca68700 1 -- start start 2023-11-15T04:34:30.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.635+0000 7f8fcca68700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 0x7f8fc810b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:30.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.635+0000 7f8fcca68700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8107210 con 0x7f8fc80f5410 2023-11-15T04:34:30.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.635+0000 7f8fc659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 0x7f8fc810b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:30.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.635+0000 7f8fc659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 0x7f8fc810b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56674/0 (socket says 172.21.15.190:56674) 2023-11-15T04:34:30.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.635+0000 7f8fc659c700 1 -- 172.21.15.190:0/63357054 learned_addr learned my addr 172.21.15.190:0/63357054 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:30.640 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.635+0000 7f8fc659c700 1 -- 172.21.15.190:0/63357054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb8005040 con 0x7f8fc80f5410 2023-11-15T04:34:30.640 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.636+0000 7f8fc659c700 1 --2- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 0x7f8fc810b820 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f8fb8013510 tx=0x7f8fb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:30.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.636+0000 7f8fbeffd700 1 -- 172.21.15.190:0/63357054 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb8009320 con 0x7f8fc80f5410 2023-11-15T04:34:30.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.636+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc8060220 con 0x7f8fc80f5410 2023-11-15T04:34:30.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.636+0000 7f8fbeffd700 1 -- 172.21.15.190:0/63357054 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8fb8013f00 con 0x7f8fc80f5410 2023-11-15T04:34:30.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.636+0000 7f8fbeffd700 1 -- 172.21.15.190:0/63357054 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb8014d90 con 0x7f8fc80f5410 2023-11-15T04:34:30.641 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.636+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc810bd10 con 0x7f8fc80f5410 2023-11-15T04:34:30.642 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.636+0000 7f8fbeffd700 1 -- 172.21.15.190:0/63357054 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8fb80095f0 con 0x7f8fc80f5410 2023-11-15T04:34:30.644 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.636+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc8045ad0 con 0x7f8fc80f5410 2023-11-15T04:34:30.645 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.637+0000 7f8fbeffd700 1 --2- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8fb402ff60 0x7f8fb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:30.645 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.637+0000 7f8fbeffd700 1 -- 172.21.15.190:0/63357054 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8fb80410c0 con 0x7f8fc80f5410 2023-11-15T04:34:30.645 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.637+0000 7f8fc5d9b700 1 --2- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8fb402ff60 0x7f8fb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:30.645 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.637+0000 7f8fc5d9b700 1 --2- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8fb402ff60 0x7f8fb4032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8fb00096a0 tx=0x7f8fb0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:30.645 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.642+0000 7f8fbeffd700 1 -- 172.21.15.190:0/63357054 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8fb801d7f0 con 0x7f8fc80f5410 2023-11-15T04:34:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.848+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8fc80fae60 con 0x7f8fc80f5410 2023-11-15T04:34:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.849+0000 7f8fbeffd700 1 -- 172.21.15.190:0/63357054 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8fb8010230 con 0x7f8fc80f5410 2023-11-15T04:34:30.850 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:30.851 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8fb402ff60 msgr2=0x7f8fb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 --2- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8fb402ff60 0x7f8fb4032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8fb00096a0 tx=0x7f8fb0006b40).stop 2023-11-15T04:34:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 msgr2=0x7f8fc810b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 --2- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 0x7f8fc810b820 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f8fb8013510 tx=0x7f8fb800b040).stop 2023-11-15T04:34:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 shutdown_connections 2023-11-15T04:34:30.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 --2- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8fb402ff60 0x7f8fb4032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:30.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 --2- 172.21.15.190:0/63357054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fc80f5410 0x7f8fc810b820 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:30.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 >> 172.21.15.190:0/63357054 conn(0x7f8fc80f0b70 msgr2=0x7f8fc80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:30.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 shutdown_connections 2023-11-15T04:34:30.853 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:30.851+0000 7f8fcca68700 1 -- 172.21.15.190:0/63357054 wait complete. 2023-11-15T04:34:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:31.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:30.850594+0000 mon.smithi047 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.190:0/63357054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:32.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:32.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:32.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:31 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:31.107869+0000 mgr.smithi047.zgysuz (mgr.14164) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:33.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.357+0000 7ff143395700 1 -- 172.21.15.190:0/1984219661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 msgr2=0x7ff13c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:33.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.357+0000 7ff143395700 1 --2- 172.21.15.190:0/1984219661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 0x7ff13c0f55c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7ff138009fb0 tx=0x7ff138009b30).stop 2023-11-15T04:34:33.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.357+0000 7ff143395700 1 -- 172.21.15.190:0/1984219661 shutdown_connections 2023-11-15T04:34:33.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.357+0000 7ff143395700 1 --2- 172.21.15.190:0/1984219661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 0x7ff13c0f55c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:33.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.357+0000 7ff143395700 1 -- 172.21.15.190:0/1984219661 >> 172.21.15.190:0/1984219661 conn(0x7ff13c0f09a0 msgr2=0x7ff13c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:33.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.357+0000 7ff143395700 1 -- 172.21.15.190:0/1984219661 shutdown_connections 2023-11-15T04:34:33.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.357+0000 7ff143395700 1 -- 172.21.15.190:0/1984219661 wait complete. 2023-11-15T04:34:33.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.357+0000 7ff143395700 1 Processor -- start 2023-11-15T04:34:33.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.357+0000 7ff143395700 1 -- start start 2023-11-15T04:34:33.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.358+0000 7ff143395700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 0x7ff13c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:33.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.358+0000 7ff143395700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c1058a0 con 0x7ff13c0f51a0 2023-11-15T04:34:33.362 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.358+0000 7ff142393700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 0x7ff13c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:33.362 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.358+0000 7ff142393700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 0x7ff13c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56700/0 (socket says 172.21.15.190:56700) 2023-11-15T04:34:33.362 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.358+0000 7ff142393700 1 -- 172.21.15.190:0/309981654 learned_addr learned my addr 172.21.15.190:0/309981654 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:33.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.358+0000 7ff142393700 1 -- 172.21.15.190:0/309981654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138005040 con 0x7ff13c0f51a0 2023-11-15T04:34:33.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.358+0000 7ff142393700 1 --2- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 0x7ff13c1053b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7ff138013510 tx=0x7ff13800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:33.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.359+0000 7ff132ffd700 1 -- 172.21.15.190:0/309981654 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff138009320 con 0x7ff13c0f51a0 2023-11-15T04:34:33.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.359+0000 7ff132ffd700 1 -- 172.21.15.190:0/309981654 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff138013f00 con 0x7ff13c0f51a0 2023-11-15T04:34:33.363 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.359+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13c105aa0 con 0x7ff13c0f51a0 2023-11-15T04:34:33.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.359+0000 7ff132ffd700 1 -- 172.21.15.190:0/309981654 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff138014dc0 con 0x7ff13c0f51a0 2023-11-15T04:34:33.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.359+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13c105f30 con 0x7ff13c0f51a0 2023-11-15T04:34:33.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.359+0000 7ff132ffd700 1 -- 172.21.15.190:0/309981654 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1380095f0 con 0x7ff13c0f51a0 2023-11-15T04:34:33.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.359+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13c045ad0 con 0x7ff13c0f51a0 2023-11-15T04:34:33.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.360+0000 7ff132ffd700 1 --2- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff12802ff60 0x7ff128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:33.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.360+0000 7ff132ffd700 1 -- 172.21.15.190:0/309981654 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff138041100 con 0x7ff13c0f51a0 2023-11-15T04:34:33.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.360+0000 7ff141b92700 1 --2- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff12802ff60 0x7ff128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:33.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.360+0000 7ff141b92700 1 --2- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff12802ff60 0x7ff128032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7ff12c0096a0 tx=0x7ff12c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:33.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.364+0000 7ff132ffd700 1 -- 172.21.15.190:0/309981654 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff138014420 con 0x7ff13c0f51a0 2023-11-15T04:34:33.571 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.570+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff13c106de0 con 0x7ff13c0f51a0 2023-11-15T04:34:33.572 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.571+0000 7ff132ffd700 1 -- 172.21.15.190:0/309981654 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff138010230 con 0x7ff13c0f51a0 2023-11-15T04:34:33.572 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:33.572 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:33.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff12802ff60 msgr2=0x7ff128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:33.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 --2- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff12802ff60 0x7ff128032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7ff12c0096a0 tx=0x7ff12c006b40).stop 2023-11-15T04:34:33.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 msgr2=0x7ff13c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:33.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 --2- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 0x7ff13c1053b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7ff138013510 tx=0x7ff13800b040).stop 2023-11-15T04:34:33.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 shutdown_connections 2023-11-15T04:34:33.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 --2- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff12802ff60 0x7ff128032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:33.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 --2- 172.21.15.190:0/309981654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0f51a0 0x7ff13c1053b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:33.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 >> 172.21.15.190:0/309981654 conn(0x7ff13c0f09a0 msgr2=0x7ff13c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:33.576 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 shutdown_connections 2023-11-15T04:34:33.576 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:33.573+0000 7ff143395700 1 -- 172.21.15.190:0/309981654 wait complete. 2023-11-15T04:34:33.576 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:34.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:33.108322+0000 mgr.smithi047.zgysuz (mgr.14164) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:34.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:33.572400+0000 mon.smithi047 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.190:0/309981654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:34.957 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:34.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:36.094 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.093+0000 7f2691a26700 1 -- 172.21.15.190:0/434081856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 msgr2=0x7f268c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:36.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.093+0000 7f2691a26700 1 --2- 172.21.15.190:0/434081856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 0x7f268c0f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f2680009fb0 tx=0x7f2680009b30).stop 2023-11-15T04:34:36.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.093+0000 7f2691a26700 1 -- 172.21.15.190:0/434081856 shutdown_connections 2023-11-15T04:34:36.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.093+0000 7f2691a26700 1 --2- 172.21.15.190:0/434081856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 0x7f268c0f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:36.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.093+0000 7f2691a26700 1 -- 172.21.15.190:0/434081856 >> 172.21.15.190:0/434081856 conn(0x7f268c0f0b70 msgr2=0x7f268c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.093+0000 7f2691a26700 1 -- 172.21.15.190:0/434081856 shutdown_connections 2023-11-15T04:34:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.093+0000 7f2691a26700 1 -- 172.21.15.190:0/434081856 wait complete. 2023-11-15T04:34:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.093+0000 7f2691a26700 1 Processor -- start 2023-11-15T04:34:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.093+0000 7f2691a26700 1 -- start start 2023-11-15T04:34:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.094+0000 7f2691a26700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 0x7f268c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:36.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.094+0000 7f2691a26700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c106bf0 con 0x7f268c0f5410 2023-11-15T04:34:36.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.094+0000 7f268affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 0x7f268c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:36.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.094+0000 7f268affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 0x7f268c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56714/0 (socket says 172.21.15.190:56714) 2023-11-15T04:34:36.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.094+0000 7f268affd700 1 -- 172.21.15.190:0/4093732401 learned_addr learned my addr 172.21.15.190:0/4093732401 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:36.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.094+0000 7f268affd700 1 -- 172.21.15.190:0/4093732401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680005040 con 0x7f268c0f5410 2023-11-15T04:34:36.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.094+0000 7f268affd700 1 --2- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 0x7f268c106700 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f2680013510 tx=0x7f268000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:36.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.095+0000 7f26737fe700 1 -- 172.21.15.190:0/4093732401 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2680009320 con 0x7f268c0f5410 2023-11-15T04:34:36.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.095+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268c106df0 con 0x7f268c0f5410 2023-11-15T04:34:36.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.095+0000 7f26737fe700 1 -- 172.21.15.190:0/4093732401 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2680013f00 con 0x7f268c0f5410 2023-11-15T04:34:36.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.095+0000 7f26737fe700 1 -- 172.21.15.190:0/4093732401 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2680014d90 con 0x7f268c0f5410 2023-11-15T04:34:36.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.095+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268c107280 con 0x7f268c0f5410 2023-11-15T04:34:36.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.095+0000 7f26737fe700 1 -- 172.21.15.190:0/4093732401 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26800095f0 con 0x7f268c0f5410 2023-11-15T04:34:36.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.096+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268c045ad0 con 0x7f268c0f5410 2023-11-15T04:34:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.096+0000 7f26737fe700 1 --2- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f267402ff60 0x7f2674032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.096+0000 7f26737fe700 1 -- 172.21.15.190:0/4093732401 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26800410a0 con 0x7f268c0f5410 2023-11-15T04:34:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.096+0000 7f268a7fc700 1 --2- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f267402ff60 0x7f2674032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.097+0000 7f268a7fc700 1 --2- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f267402ff60 0x7f2674032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f267c0096a0 tx=0x7f267c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:36.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.101+0000 7f26737fe700 1 -- 172.21.15.190:0/4093732401 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f268006d2b0 con 0x7f268c0f5410 2023-11-15T04:34:36.309 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.308+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f268c0faf50 con 0x7f268c0f5410 2023-11-15T04:34:36.310 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.309+0000 7f26737fe700 1 -- 172.21.15.190:0/4093732401 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f268000f340 con 0x7f268c0f5410 2023-11-15T04:34:36.310 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:36.310 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:36.311 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f267402ff60 msgr2=0x7f2674032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:36.311 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 --2- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f267402ff60 0x7f2674032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f267c0096a0 tx=0x7f267c006b40).stop 2023-11-15T04:34:36.312 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 msgr2=0x7f268c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:36.312 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 --2- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 0x7f268c106700 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f2680013510 tx=0x7f268000b040).stop 2023-11-15T04:34:36.312 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 shutdown_connections 2023-11-15T04:34:36.312 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 --2- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f267402ff60 0x7f2674032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:36.312 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 --2- 172.21.15.190:0/4093732401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f268c0f5410 0x7f268c106700 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:36.313 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 >> 172.21.15.190:0/4093732401 conn(0x7f268c0f0b70 msgr2=0x7f268c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:36.313 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 shutdown_connections 2023-11-15T04:34:36.313 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:36.310+0000 7f2691a26700 1 -- 172.21.15.190:0/4093732401 wait complete. 2023-11-15T04:34:36.314 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:36.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:35.059638+0000 mon.smithi047 (mon.0) 245 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' 2023-11-15T04:34:36.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:34:35.061252+0000 mgr.smithi047.zgysuz (mgr.14164) 170 : cephadm [INF] Deploying daemon node-exporter.smithi047 on smithi047 2023-11-15T04:34:36.509 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:35.108602+0000 mgr.smithi047.zgysuz (mgr.14164) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:37.509 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:36.310383+0000 mon.smithi047 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.190:0/4093732401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:37.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:37.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:38.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:38 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:37.108959+0000 mgr.smithi047.zgysuz (mgr.14164) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:38.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.898+0000 7f0f9996f700 1 -- 172.21.15.190:0/517036764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f940f6510 msgr2=0x7f0f940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:38.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.898+0000 7f0f9996f700 1 --2- 172.21.15.190:0/517036764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f940f6510 0x7f0f940f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f0f88009fb0 tx=0x7f0f88009b30).stop 2023-11-15T04:34:38.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.898+0000 7f0f9996f700 1 -- 172.21.15.190:0/517036764 shutdown_connections 2023-11-15T04:34:38.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.898+0000 7f0f9996f700 1 --2- 172.21.15.190:0/517036764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f940f6510 0x7f0f940f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:38.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.898+0000 7f0f9996f700 1 -- 172.21.15.190:0/517036764 >> 172.21.15.190:0/517036764 conn(0x7f0f940f1c70 msgr2=0x7f0f940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:38.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.898+0000 7f0f9996f700 1 -- 172.21.15.190:0/517036764 shutdown_connections 2023-11-15T04:34:38.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.898+0000 7f0f9996f700 1 -- 172.21.15.190:0/517036764 wait complete. 2023-11-15T04:34:38.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.899+0000 7f0f9996f700 1 Processor -- start 2023-11-15T04:34:38.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.899+0000 7f0f9996f700 1 -- start start 2023-11-15T04:34:38.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.899+0000 7f0f9996f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f94106840 0x7f0f94106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:38.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.899+0000 7f0f9996f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f94107150 con 0x7f0f94106840 2023-11-15T04:34:38.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.899+0000 7f0f92ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f94106840 0x7f0f94106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:38.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.899+0000 7f0f92ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f94106840 0x7f0f94106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56730/0 (socket says 172.21.15.190:56730) 2023-11-15T04:34:38.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.899+0000 7f0f92ffd700 1 -- 172.21.15.190:0/2298715565 learned_addr learned my addr 172.21.15.190:0/2298715565 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:38.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.900+0000 7f0f92ffd700 1 -- 172.21.15.190:0/2298715565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f88005040 con 0x7f0f94106840 2023-11-15T04:34:38.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.900+0000 7f0f92ffd700 1 --2- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f94106840 0x7f0f94106c60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f0f88009f80 tx=0x7f0f88009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:38.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.900+0000 7f0f7b7fe700 1 -- 172.21.15.190:0/2298715565 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f880136c0 con 0x7f0f94106840 2023-11-15T04:34:38.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.900+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f94107350 con 0x7f0f94106840 2023-11-15T04:34:38.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.900+0000 7f0f7b7fe700 1 -- 172.21.15.190:0/2298715565 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f88013e70 con 0x7f0f94106840 2023-11-15T04:34:38.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.900+0000 7f0f7b7fe700 1 -- 172.21.15.190:0/2298715565 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f88014420 con 0x7f0f94106840 2023-11-15T04:34:38.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.900+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f94107ea0 con 0x7f0f94106840 2023-11-15T04:34:38.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.901+0000 7f0f7b7fe700 1 -- 172.21.15.190:0/2298715565 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f88025060 con 0x7f0f94106840 2023-11-15T04:34:38.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.901+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f94059100 con 0x7f0f94106840 2023-11-15T04:34:38.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.901+0000 7f0f7b7fe700 1 --2- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0f7c02ff60 0x7f0f7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:38.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.901+0000 7f0f7b7fe700 1 -- 172.21.15.190:0/2298715565 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f88040e00 con 0x7f0f94106840 2023-11-15T04:34:38.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.901+0000 7f0f927fc700 1 --2- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0f7c02ff60 0x7f0f7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:38.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.902+0000 7f0f927fc700 1 --2- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0f7c02ff60 0x7f0f7c032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f0f840096a0 tx=0x7f0f84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:38.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:38.906+0000 7f0f7b7fe700 1 -- 172.21.15.190:0/2298715565 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f8801cd10 con 0x7f0f94106840 2023-11-15T04:34:39.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.110+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f94045ad0 con 0x7f0f94106840 2023-11-15T04:34:39.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.110+0000 7f0f7b7fe700 1 -- 172.21.15.190:0/2298715565 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f8802d2d0 con 0x7f0f94106840 2023-11-15T04:34:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:39.112 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:39.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0f7c02ff60 msgr2=0x7f0f7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:39.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 --2- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0f7c02ff60 0x7f0f7c032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f0f840096a0 tx=0x7f0f84006b40).stop 2023-11-15T04:34:39.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f94106840 msgr2=0x7f0f94106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:39.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 --2- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f94106840 0x7f0f94106c60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f0f88009f80 tx=0x7f0f88009b30).stop 2023-11-15T04:34:39.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 shutdown_connections 2023-11-15T04:34:39.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 --2- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0f7c02ff60 0x7f0f7c032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:39.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 --2- 172.21.15.190:0/2298715565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f94106840 0x7f0f94106c60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:39.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 >> 172.21.15.190:0/2298715565 conn(0x7f0f940f1c70 msgr2=0x7f0f940f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:39.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 shutdown_connections 2023-11-15T04:34:39.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:39.112+0000 7f0f9996f700 1 -- 172.21.15.190:0/2298715565 wait complete. 2023-11-15T04:34:39.116 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:39.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:39 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:39.111900+0000 mon.smithi047 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.190:0/2298715565' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:40.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:40.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:40.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:39.109386+0000 mgr.smithi047.zgysuz (mgr.14164) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:41.644 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.642+0000 7f3228829700 1 -- 172.21.15.190:0/2029717112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 msgr2=0x7f32240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:41.644 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.642+0000 7f3228829700 1 --2- 172.21.15.190:0/2029717112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 0x7f32240f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f320c009fb0 tx=0x7f320c009b30).stop 2023-11-15T04:34:41.644 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.642+0000 7f3228829700 1 -- 172.21.15.190:0/2029717112 shutdown_connections 2023-11-15T04:34:41.645 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.642+0000 7f3228829700 1 --2- 172.21.15.190:0/2029717112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 0x7f32240f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:41.645 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.642+0000 7f3228829700 1 -- 172.21.15.190:0/2029717112 >> 172.21.15.190:0/2029717112 conn(0x7f32240f0b70 msgr2=0x7f32240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:41.645 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.642+0000 7f3228829700 1 -- 172.21.15.190:0/2029717112 shutdown_connections 2023-11-15T04:34:41.645 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.642+0000 7f3228829700 1 -- 172.21.15.190:0/2029717112 wait complete. 2023-11-15T04:34:41.646 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.643+0000 7f3228829700 1 Processor -- start 2023-11-15T04:34:41.646 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.643+0000 7f3228829700 1 -- start start 2023-11-15T04:34:41.646 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.643+0000 7f3228829700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 0x7f32241093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:41.646 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.643+0000 7f3228829700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3224060220 con 0x7f32240f5410 2023-11-15T04:34:41.646 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.643+0000 7f322259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 0x7f32241093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:41.647 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.643+0000 7f322259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 0x7f32241093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:50720/0 (socket says 172.21.15.190:50720) 2023-11-15T04:34:41.647 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.643+0000 7f322259c700 1 -- 172.21.15.190:0/3777586534 learned_addr learned my addr 172.21.15.190:0/3777586534 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:41.648 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.644+0000 7f322259c700 1 -- 172.21.15.190:0/3777586534 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c005040 con 0x7f32240f5410 2023-11-15T04:34:41.648 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.644+0000 7f322259c700 1 --2- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 0x7f32241093f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f320c013510 tx=0x7f320c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:41.649 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.644+0000 7f321affd700 1 -- 172.21.15.190:0/3777586534 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f320c009320 con 0x7f32240f5410 2023-11-15T04:34:41.649 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.644+0000 7f321affd700 1 -- 172.21.15.190:0/3777586534 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f320c013f00 con 0x7f32240f5410 2023-11-15T04:34:41.649 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.644+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3224060420 con 0x7f32240f5410 2023-11-15T04:34:41.649 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.644+0000 7f321affd700 1 -- 172.21.15.190:0/3777586534 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f320c014d90 con 0x7f32240f5410 2023-11-15T04:34:41.649 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.644+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3224109aa0 con 0x7f32240f5410 2023-11-15T04:34:41.650 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.645+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3224059100 con 0x7f32240f5410 2023-11-15T04:34:41.651 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.645+0000 7f321affd700 1 -- 172.21.15.190:0/3777586534 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f320c0095f0 con 0x7f32240f5410 2023-11-15T04:34:41.651 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.645+0000 7f321affd700 1 --2- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f321002ff60 0x7f3210032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:41.651 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.645+0000 7f321affd700 1 -- 172.21.15.190:0/3777586534 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f320c041000 con 0x7f32240f5410 2023-11-15T04:34:41.651 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.645+0000 7f3221d9b700 1 --2- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f321002ff60 0x7f3210032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:41.652 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.646+0000 7f3221d9b700 1 --2- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f321002ff60 0x7f3210032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f32140096a0 tx=0x7f3214006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:41.652 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.649+0000 7f321affd700 1 -- 172.21.15.190:0/3777586534 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f320c01d7f0 con 0x7f32240f5410 2023-11-15T04:34:41.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.857+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3224045ad0 con 0x7f32240f5410 2023-11-15T04:34:41.859 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.857+0000 7f321affd700 1 -- 172.21.15.190:0/3777586534 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f320c010230 con 0x7f32240f5410 2023-11-15T04:34:41.860 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:41.860 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:41.861 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f321002ff60 msgr2=0x7f3210032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:41.863 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 --2- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f321002ff60 0x7f3210032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f32140096a0 tx=0x7f3214006b40).stop 2023-11-15T04:34:41.863 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 msgr2=0x7f32241093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:41.863 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 --2- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 0x7f32241093f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f320c013510 tx=0x7f320c00b040).stop 2023-11-15T04:34:41.863 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 shutdown_connections 2023-11-15T04:34:41.864 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 --2- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f321002ff60 0x7f3210032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:41.864 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 --2- 172.21.15.190:0/3777586534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32240f5410 0x7f32241093f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:41.864 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 >> 172.21.15.190:0/3777586534 conn(0x7f32240f0b70 msgr2=0x7f32240f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:41.864 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 shutdown_connections 2023-11-15T04:34:41.864 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:41.859+0000 7f3228829700 1 -- 172.21.15.190:0/3777586534 wait complete. 2023-11-15T04:34:41.865 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:42.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:41.109789+0000 mgr.smithi047.zgysuz (mgr.14164) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:42.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:41.859054+0000 mon.smithi047 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.190:0/3777586534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:43.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:43.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:44.415 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.413+0000 7f7c4e388700 1 -- 172.21.15.190:0/1252129262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 msgr2=0x7f7c480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:44.415 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.413+0000 7f7c4e388700 1 --2- 172.21.15.190:0/1252129262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 0x7f7c480f6930 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f7c38009fb0 tx=0x7f7c38009b30).stop 2023-11-15T04:34:44.415 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.413+0000 7f7c4e388700 1 -- 172.21.15.190:0/1252129262 shutdown_connections 2023-11-15T04:34:44.416 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.413+0000 7f7c4e388700 1 --2- 172.21.15.190:0/1252129262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 0x7f7c480f6930 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:44.416 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.413+0000 7f7c4e388700 1 -- 172.21.15.190:0/1252129262 >> 172.21.15.190:0/1252129262 conn(0x7f7c480f1c70 msgr2=0x7f7c480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:44.416 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.413+0000 7f7c4e388700 1 -- 172.21.15.190:0/1252129262 shutdown_connections 2023-11-15T04:34:44.416 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.413+0000 7f7c4e388700 1 -- 172.21.15.190:0/1252129262 wait complete. 2023-11-15T04:34:44.416 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.414+0000 7f7c4e388700 1 Processor -- start 2023-11-15T04:34:44.417 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.414+0000 7f7c4e388700 1 -- start start 2023-11-15T04:34:44.417 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.414+0000 7f7c4e388700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 0x7f7c48106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:44.417 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.414+0000 7f7c4e388700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48106c20 con 0x7f7c480f6510 2023-11-15T04:34:44.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.414+0000 7f7c47fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 0x7f7c48106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:44.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.414+0000 7f7c47fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 0x7f7c48106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:50728/0 (socket says 172.21.15.190:50728) 2023-11-15T04:34:44.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.414+0000 7f7c47fff700 1 -- 172.21.15.190:0/1953581439 learned_addr learned my addr 172.21.15.190:0/1953581439 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:44.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.415+0000 7f7c47fff700 1 -- 172.21.15.190:0/1953581439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c38005040 con 0x7f7c480f6510 2023-11-15T04:34:44.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.415+0000 7f7c47fff700 1 --2- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 0x7f7c48106730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f7c38013510 tx=0x7f7c3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:44.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.415+0000 7f7c44ff9700 1 -- 172.21.15.190:0/1953581439 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c38009320 con 0x7f7c480f6510 2023-11-15T04:34:44.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.415+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c48106e20 con 0x7f7c480f6510 2023-11-15T04:34:44.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.415+0000 7f7c44ff9700 1 -- 172.21.15.190:0/1953581439 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c38013f00 con 0x7f7c480f6510 2023-11-15T04:34:44.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.415+0000 7f7c44ff9700 1 -- 172.21.15.190:0/1953581439 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c38014d90 con 0x7f7c480f6510 2023-11-15T04:34:44.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.415+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c481072b0 con 0x7f7c480f6510 2023-11-15T04:34:44.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.416+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c48045ad0 con 0x7f7c480f6510 2023-11-15T04:34:44.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.416+0000 7f7c44ff9700 1 -- 172.21.15.190:0/1953581439 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c380095f0 con 0x7f7c480f6510 2023-11-15T04:34:44.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.416+0000 7f7c44ff9700 1 --2- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c3402ff60 0x7f7c34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:44.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.416+0000 7f7c44ff9700 1 -- 172.21.15.190:0/1953581439 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c38040fc0 con 0x7f7c480f6510 2023-11-15T04:34:44.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.416+0000 7f7c477fe700 1 --2- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c3402ff60 0x7f7c34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:44.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.417+0000 7f7c477fe700 1 --2- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c3402ff60 0x7f7c34032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f7c300096a0 tx=0x7f7c30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:44.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.421+0000 7f7c44ff9700 1 -- 172.21.15.190:0/1953581439 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c38014420 con 0x7f7c480f6510 2023-11-15T04:34:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:43.110187+0000 mgr.smithi047.zgysuz (mgr.14164) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:44.626 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.625+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c480fbf60 con 0x7f7c480f6510 2023-11-15T04:34:44.627 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.626+0000 7f7c44ff9700 1 -- 172.21.15.190:0/1953581439 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c38010230 con 0x7f7c480f6510 2023-11-15T04:34:44.627 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:44.627 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:44.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c3402ff60 msgr2=0x7f7c34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:44.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 --2- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c3402ff60 0x7f7c34032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f7c300096a0 tx=0x7f7c30006b40).stop 2023-11-15T04:34:44.629 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 msgr2=0x7f7c48106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:44.629 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 --2- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 0x7f7c48106730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f7c38013510 tx=0x7f7c3800b040).stop 2023-11-15T04:34:44.629 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 shutdown_connections 2023-11-15T04:34:44.629 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 --2- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c3402ff60 0x7f7c34032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:44.629 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 --2- 172.21.15.190:0/1953581439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c480f6510 0x7f7c48106730 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:44.630 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 >> 172.21.15.190:0/1953581439 conn(0x7f7c480f1c70 msgr2=0x7f7c480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:44.630 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 shutdown_connections 2023-11-15T04:34:44.630 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:44.627+0000 7f7c4e388700 1 -- 172.21.15.190:0/1953581439 wait complete. 2023-11-15T04:34:44.630 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:45 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:44.627442+0000 mon.smithi047 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.190:0/1953581439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:46.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:46.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:45.110571+0000 mgr.smithi047.zgysuz (mgr.14164) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:47.252 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.250+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2848497585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 msgr2=0x7fc2380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.250+0000 7fc23f0e7700 1 --2- 172.21.15.190:0/2848497585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 0x7fc2380f5830 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fc22c009fb0 tx=0x7fc22c009b30).stop 2023-11-15T04:34:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.250+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2848497585 shutdown_connections 2023-11-15T04:34:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.250+0000 7fc23f0e7700 1 --2- 172.21.15.190:0/2848497585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 0x7fc2380f5830 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:47.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.250+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2848497585 >> 172.21.15.190:0/2848497585 conn(0x7fc2380f0b70 msgr2=0x7fc2380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:47.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.250+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2848497585 shutdown_connections 2023-11-15T04:34:47.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.250+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2848497585 wait complete. 2023-11-15T04:34:47.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.250+0000 7fc23f0e7700 1 Processor -- start 2023-11-15T04:34:47.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.250+0000 7fc23f0e7700 1 -- start start 2023-11-15T04:34:47.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.251+0000 7fc23f0e7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 0x7fc2381088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:47.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.251+0000 7fc23f0e7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc238108dc0 con 0x7fc2380f5410 2023-11-15T04:34:47.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.251+0000 7fc23ce83700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 0x7fc2381088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:47.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.251+0000 7fc23ce83700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 0x7fc2381088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:50760/0 (socket says 172.21.15.190:50760) 2023-11-15T04:34:47.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.251+0000 7fc23ce83700 1 -- 172.21.15.190:0/2997917017 learned_addr learned my addr 172.21.15.190:0/2997917017 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:47.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.251+0000 7fc23ce83700 1 -- 172.21.15.190:0/2997917017 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c005040 con 0x7fc2380f5410 2023-11-15T04:34:47.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.251+0000 7fc23ce83700 1 --2- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 0x7fc2381088d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fc22c013510 tx=0x7fc22c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:47.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.252+0000 7fc2357fa700 1 -- 172.21.15.190:0/2997917017 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc22c009320 con 0x7fc2380f5410 2023-11-15T04:34:47.256 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.252+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc238108fc0 con 0x7fc2380f5410 2023-11-15T04:34:47.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.252+0000 7fc2357fa700 1 -- 172.21.15.190:0/2997917017 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc22c013f00 con 0x7fc2380f5410 2023-11-15T04:34:47.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.252+0000 7fc2357fa700 1 -- 172.21.15.190:0/2997917017 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc22c014d90 con 0x7fc2380f5410 2023-11-15T04:34:47.257 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.252+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc238109450 con 0x7fc2380f5410 2023-11-15T04:34:47.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.252+0000 7fc2357fa700 1 -- 172.21.15.190:0/2997917017 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc22c014420 con 0x7fc2380f5410 2023-11-15T04:34:47.259 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.252+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc238045ad0 con 0x7fc2380f5410 2023-11-15T04:34:47.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.253+0000 7fc2357fa700 1 --2- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc22802ff60 0x7fc228032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:47.260 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.253+0000 7fc2357fa700 1 -- 172.21.15.190:0/2997917017 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc22c041080 con 0x7fc2380f5410 2023-11-15T04:34:47.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.253+0000 7fc237fff700 1 --2- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc22802ff60 0x7fc228032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:47.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.253+0000 7fc237fff700 1 --2- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc22802ff60 0x7fc228032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc2240096a0 tx=0x7fc224006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:47.261 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.258+0000 7fc2357fa700 1 -- 172.21.15.190:0/2997917017 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc22c0095f0 con 0x7fc2380f5410 2023-11-15T04:34:47.464 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.463+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc23810a300 con 0x7fc2380f5410 2023-11-15T04:34:47.464 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.464+0000 7fc2357fa700 1 -- 172.21.15.190:0/2997917017 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc22c01db70 con 0x7fc2380f5410 2023-11-15T04:34:47.465 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:47.465 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc22802ff60 msgr2=0x7fc228032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 --2- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc22802ff60 0x7fc228032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc2240096a0 tx=0x7fc224006b40).stop 2023-11-15T04:34:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 msgr2=0x7fc2381088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:47.467 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 --2- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 0x7fc2381088d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fc22c013510 tx=0x7fc22c00b040).stop 2023-11-15T04:34:47.467 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 shutdown_connections 2023-11-15T04:34:47.467 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 --2- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc22802ff60 0x7fc228032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:47.467 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 --2- 172.21.15.190:0/2997917017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2380f5410 0x7fc2381088d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:47.467 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 >> 172.21.15.190:0/2997917017 conn(0x7fc2380f0b70 msgr2=0x7fc23805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:47.468 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 shutdown_connections 2023-11-15T04:34:47.468 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:47.465+0000 7fc23f0e7700 1 -- 172.21.15.190:0/2997917017 wait complete. 2023-11-15T04:34:47.468 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:47.110827+0000 mgr.smithi047.zgysuz (mgr.14164) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:47.465220+0000 mon.smithi047 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.190:0/2997917017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:48.842 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:48.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:49.931 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.931+0000 7fdc84a02700 1 -- 172.21.15.190:0/2292995415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 msgr2=0x7fdc800f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:49.932 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.931+0000 7fdc84a02700 1 --2- 172.21.15.190:0/2292995415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 0x7fdc800f66c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fdc74009fb0 tx=0x7fdc74009b30).stop 2023-11-15T04:34:49.932 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.931+0000 7fdc84a02700 1 -- 172.21.15.190:0/2292995415 shutdown_connections 2023-11-15T04:34:49.932 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.931+0000 7fdc84a02700 1 --2- 172.21.15.190:0/2292995415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 0x7fdc800f66c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:49.932 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.931+0000 7fdc84a02700 1 -- 172.21.15.190:0/2292995415 >> 172.21.15.190:0/2292995415 conn(0x7fdc800f1aa0 msgr2=0x7fdc800f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:49.933 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.931+0000 7fdc84a02700 1 -- 172.21.15.190:0/2292995415 shutdown_connections 2023-11-15T04:34:49.933 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.931+0000 7fdc84a02700 1 -- 172.21.15.190:0/2292995415 wait complete. 2023-11-15T04:34:49.933 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.932+0000 7fdc84a02700 1 Processor -- start 2023-11-15T04:34:49.933 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.932+0000 7fdc84a02700 1 -- start start 2023-11-15T04:34:49.933 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.932+0000 7fdc84a02700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 0x7fdc8006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:49.934 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.932+0000 7fdc84a02700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc80069e20 con 0x7fdc800f62a0 2023-11-15T04:34:49.935 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.932+0000 7fdc7ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 0x7fdc8006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:49.935 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.932+0000 7fdc7ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 0x7fdc8006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:50776/0 (socket says 172.21.15.190:50776) 2023-11-15T04:34:49.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.932+0000 7fdc7ed9d700 1 -- 172.21.15.190:0/163193642 learned_addr learned my addr 172.21.15.190:0/163193642 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:49.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.932+0000 7fdc7ed9d700 1 -- 172.21.15.190:0/163193642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc74005040 con 0x7fdc800f62a0 2023-11-15T04:34:49.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.932+0000 7fdc7ed9d700 1 --2- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 0x7fdc8006b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fdc74013510 tx=0x7fdc7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:49.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.933+0000 7fdc677fe700 1 -- 172.21.15.190:0/163193642 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc74009320 con 0x7fdc800f62a0 2023-11-15T04:34:49.936 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.933+0000 7fdc677fe700 1 -- 172.21.15.190:0/163193642 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc74013f00 con 0x7fdc800f62a0 2023-11-15T04:34:49.937 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.933+0000 7fdc677fe700 1 -- 172.21.15.190:0/163193642 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc74014dc0 con 0x7fdc800f62a0 2023-11-15T04:34:49.937 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.933+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc80069fc0 con 0x7fdc800f62a0 2023-11-15T04:34:49.937 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.933+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8006a450 con 0x7fdc800f62a0 2023-11-15T04:34:49.937 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.933+0000 7fdc677fe700 1 -- 172.21.15.190:0/163193642 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc7401d3d0 con 0x7fdc800f62a0 2023-11-15T04:34:49.937 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.934+0000 7fdc677fe700 1 --2- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc6802ff60 0x7fdc68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:49.938 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.934+0000 7fdc677fe700 1 -- 172.21.15.190:0/163193642 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc74041040 con 0x7fdc800f62a0 2023-11-15T04:34:49.938 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.934+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc80059100 con 0x7fdc800f62a0 2023-11-15T04:34:49.938 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.934+0000 7fdc7e59c700 1 --2- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc6802ff60 0x7fdc68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:49.938 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.934+0000 7fdc7e59c700 1 --2- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc6802ff60 0x7fdc68032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fdc700096a0 tx=0x7fdc70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:49.939 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:49.939+0000 7fdc677fe700 1 -- 172.21.15.190:0/163193642 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc74012920 con 0x7fdc800f62a0 2023-11-15T04:34:50.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.145+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc8006ab50 con 0x7fdc800f62a0 2023-11-15T04:34:50.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.145+0000 7fdc677fe700 1 -- 172.21.15.190:0/163193642 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc7402b020 con 0x7fdc800f62a0 2023-11-15T04:34:50.146 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:50.146 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:50.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc6802ff60 msgr2=0x7fdc68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:50.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 --2- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc6802ff60 0x7fdc68032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fdc700096a0 tx=0x7fdc70006b40).stop 2023-11-15T04:34:50.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 msgr2=0x7fdc8006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:50.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 --2- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 0x7fdc8006b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fdc74013510 tx=0x7fdc7400b040).stop 2023-11-15T04:34:50.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 shutdown_connections 2023-11-15T04:34:50.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 --2- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fdc6802ff60 0x7fdc68032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:50.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 --2- 172.21.15.190:0/163193642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc800f62a0 0x7fdc8006b9a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:50.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 >> 172.21.15.190:0/163193642 conn(0x7fdc800f1aa0 msgr2=0x7fdc800f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:50.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 shutdown_connections 2023-11-15T04:34:50.150 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:50.147+0000 7fdc84a02700 1 -- 172.21.15.190:0/163193642 wait complete. 2023-11-15T04:34:50.150 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:50.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:50 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:49.111166+0000 mgr.smithi047.zgysuz (mgr.14164) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:50.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:50 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:50.145948+0000 mon.smithi047 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.190:0/163193642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:51.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:51.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:51.111493+0000 mgr.smithi047.zgysuz (mgr.14164) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:52.679 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.679+0000 7f0e2df06700 1 -- 172.21.15.190:0/3774328032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 msgr2=0x7f0e280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:52.679 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.679+0000 7f0e2df06700 1 --2- 172.21.15.190:0/3774328032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 0x7f0e280f6930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f0e1c009fb0 tx=0x7f0e1c009b30).stop 2023-11-15T04:34:52.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.679+0000 7f0e2df06700 1 -- 172.21.15.190:0/3774328032 shutdown_connections 2023-11-15T04:34:52.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.679+0000 7f0e2df06700 1 --2- 172.21.15.190:0/3774328032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 0x7f0e280f6930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:52.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.679+0000 7f0e2df06700 1 -- 172.21.15.190:0/3774328032 >> 172.21.15.190:0/3774328032 conn(0x7f0e280f1c70 msgr2=0x7f0e280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:52.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.679+0000 7f0e2df06700 1 -- 172.21.15.190:0/3774328032 shutdown_connections 2023-11-15T04:34:52.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.679+0000 7f0e2df06700 1 -- 172.21.15.190:0/3774328032 wait complete. 2023-11-15T04:34:52.681 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.680+0000 7f0e2df06700 1 Processor -- start 2023-11-15T04:34:52.682 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.680+0000 7f0e2df06700 1 -- start start 2023-11-15T04:34:52.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.680+0000 7f0e2df06700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 0x7f0e28106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:52.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.680+0000 7f0e2df06700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e28106c20 con 0x7f0e280f6510 2023-11-15T04:34:52.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.680+0000 7f0e277fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 0x7f0e28106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:52.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.680+0000 7f0e277fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 0x7f0e28106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:54712/0 (socket says 172.21.15.190:54712) 2023-11-15T04:34:52.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.680+0000 7f0e277fe700 1 -- 172.21.15.190:0/2938352606 learned_addr learned my addr 172.21.15.190:0/2938352606 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:52.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.680+0000 7f0e277fe700 1 -- 172.21.15.190:0/2938352606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c005040 con 0x7f0e280f6510 2023-11-15T04:34:52.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.681+0000 7f0e277fe700 1 --2- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 0x7f0e28106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f0e1c014000 tx=0x7f0e1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:52.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.681+0000 7f0e0ffff700 1 -- 172.21.15.190:0/2938352606 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1c0095f0 con 0x7f0e280f6510 2023-11-15T04:34:52.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.681+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e28106e20 con 0x7f0e280f6510 2023-11-15T04:34:52.685 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.681+0000 7f0e0ffff700 1 -- 172.21.15.190:0/2938352606 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e1c007e70 con 0x7f0e280f6510 2023-11-15T04:34:52.685 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.681+0000 7f0e0ffff700 1 -- 172.21.15.190:0/2938352606 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1c014ea0 con 0x7f0e280f6510 2023-11-15T04:34:52.685 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.681+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e281072b0 con 0x7f0e280f6510 2023-11-15T04:34:52.685 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.681+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e28045ad0 con 0x7f0e280f6510 2023-11-15T04:34:52.686 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.682+0000 7f0e0ffff700 1 -- 172.21.15.190:0/2938352606 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e1c025070 con 0x7f0e280f6510 2023-11-15T04:34:52.686 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.682+0000 7f0e0ffff700 1 --2- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0e1002ff60 0x7f0e10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:52.686 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.682+0000 7f0e0ffff700 1 -- 172.21.15.190:0/2938352606 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e1c040940 con 0x7f0e280f6510 2023-11-15T04:34:52.686 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.682+0000 7f0e26ffd700 1 --2- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0e1002ff60 0x7f0e10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:52.686 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.682+0000 7f0e26ffd700 1 --2- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0e1002ff60 0x7f0e10032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0e180096a0 tx=0x7f0e18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:52.687 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.687+0000 7f0e0ffff700 1 -- 172.21.15.190:0/2938352606 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e1c0079c0 con 0x7f0e280f6510 2023-11-15T04:34:52.894 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.894+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e280fbf60 con 0x7f0e280f6510 2023-11-15T04:34:52.895 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.895+0000 7f0e0ffff700 1 -- 172.21.15.190:0/2938352606 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e1c00f300 con 0x7f0e280f6510 2023-11-15T04:34:52.896 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:52.896 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:52.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0e1002ff60 msgr2=0x7f0e10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:52.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 --2- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0e1002ff60 0x7f0e10032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0e180096a0 tx=0x7f0e18006b40).stop 2023-11-15T04:34:52.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 msgr2=0x7f0e28106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:52.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 --2- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 0x7f0e28106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f0e1c014000 tx=0x7f0e1c00b040).stop 2023-11-15T04:34:52.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 shutdown_connections 2023-11-15T04:34:52.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 --2- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0e1002ff60 0x7f0e10032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:52.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 --2- 172.21.15.190:0/2938352606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e280f6510 0x7f0e28106730 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:52.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 >> 172.21.15.190:0/2938352606 conn(0x7f0e280f1c70 msgr2=0x7f0e280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:52.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 shutdown_connections 2023-11-15T04:34:52.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:52.897+0000 7f0e2df06700 1 -- 172.21.15.190:0/2938352606 wait complete. 2023-11-15T04:34:52.899 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:53.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:53 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:52.895538+0000 mon.smithi047 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.190:0/2938352606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:54.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:54.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:53.111834+0000 mgr.smithi047.zgysuz (mgr.14164) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:55.422 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.421+0000 7efeeffff700 1 -- 172.21.15.190:0/4225015274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 msgr2=0x7efef00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:55.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.421+0000 7efeeffff700 1 --2- 172.21.15.190:0/4225015274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 0x7efef00f55c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7efee0009fb0 tx=0x7efee0009b30).stop 2023-11-15T04:34:55.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.421+0000 7efeeffff700 1 -- 172.21.15.190:0/4225015274 shutdown_connections 2023-11-15T04:34:55.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.421+0000 7efeeffff700 1 --2- 172.21.15.190:0/4225015274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 0x7efef00f55c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:55.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.421+0000 7efeeffff700 1 -- 172.21.15.190:0/4225015274 >> 172.21.15.190:0/4225015274 conn(0x7efef00f09a0 msgr2=0x7efef00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:55.423 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.421+0000 7efeeffff700 1 -- 172.21.15.190:0/4225015274 shutdown_connections 2023-11-15T04:34:55.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.421+0000 7efeeffff700 1 -- 172.21.15.190:0/4225015274 wait complete. 2023-11-15T04:34:55.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.421+0000 7efeeffff700 1 Processor -- start 2023-11-15T04:34:55.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.421+0000 7efeeffff700 1 -- start start 2023-11-15T04:34:55.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.422+0000 7efeeffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 0x7efef01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:55.424 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.422+0000 7efeeffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef0060240 con 0x7efef00f51a0 2023-11-15T04:34:55.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.422+0000 7efeeeffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 0x7efef01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:55.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.422+0000 7efeeeffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 0x7efef01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:54726/0 (socket says 172.21.15.190:54726) 2023-11-15T04:34:55.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.422+0000 7efeeeffd700 1 -- 172.21.15.190:0/263905613 learned_addr learned my addr 172.21.15.190:0/263905613 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:55.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.422+0000 7efeeeffd700 1 -- 172.21.15.190:0/263905613 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee0005040 con 0x7efef00f51a0 2023-11-15T04:34:55.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.422+0000 7efeeeffd700 1 --2- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 0x7efef01091b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7efee0013510 tx=0x7efee000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:55.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.422+0000 7efed77fe700 1 -- 172.21.15.190:0/263905613 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee0009320 con 0x7efef00f51a0 2023-11-15T04:34:55.426 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.422+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef0060440 con 0x7efef00f51a0 2023-11-15T04:34:55.426 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.423+0000 7efed77fe700 1 -- 172.21.15.190:0/263905613 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efee0013f00 con 0x7efef00f51a0 2023-11-15T04:34:55.426 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.423+0000 7efed77fe700 1 -- 172.21.15.190:0/263905613 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee0014dc0 con 0x7efef00f51a0 2023-11-15T04:34:55.426 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.423+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef0109860 con 0x7efef00f51a0 2023-11-15T04:34:55.427 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.423+0000 7efed77fe700 1 -- 172.21.15.190:0/263905613 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efee00095f0 con 0x7efef00f51a0 2023-11-15T04:34:55.427 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.423+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef0045ad0 con 0x7efef00f51a0 2023-11-15T04:34:55.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.424+0000 7efed77fe700 1 --2- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efed802ff60 0x7efed8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:55.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.424+0000 7efed77fe700 1 -- 172.21.15.190:0/263905613 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efee0041000 con 0x7efef00f51a0 2023-11-15T04:34:55.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.424+0000 7efeee7fc700 1 --2- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efed802ff60 0x7efed8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:55.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.424+0000 7efeee7fc700 1 --2- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efed802ff60 0x7efed8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7efee40096a0 tx=0x7efee4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:55.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.428+0000 7efed77fe700 1 -- 172.21.15.190:0/263905613 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efee0014420 con 0x7efef00f51a0 2023-11-15T04:34:55.636 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.636+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efef010a080 con 0x7efef00f51a0 2023-11-15T04:34:55.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.637+0000 7efed77fe700 1 -- 172.21.15.190:0/263905613 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efee0010230 con 0x7efef00f51a0 2023-11-15T04:34:55.637 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:55.637 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:55.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efed802ff60 msgr2=0x7efed8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:55.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 --2- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efed802ff60 0x7efed8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7efee40096a0 tx=0x7efee4006b40).stop 2023-11-15T04:34:55.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 msgr2=0x7efef01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:55.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 --2- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 0x7efef01091b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7efee0013510 tx=0x7efee000b040).stop 2023-11-15T04:34:55.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 shutdown_connections 2023-11-15T04:34:55.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 --2- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efed802ff60 0x7efed8032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:55.639 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 --2- 172.21.15.190:0/263905613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efef00f51a0 0x7efef01091b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:55.640 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 >> 172.21.15.190:0/263905613 conn(0x7efef00f09a0 msgr2=0x7efef00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:55.640 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 shutdown_connections 2023-11-15T04:34:55.640 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:55.638+0000 7efeeffff700 1 -- 172.21.15.190:0/263905613 wait complete. 2023-11-15T04:34:55.641 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:55.112120+0000 mgr.smithi047.zgysuz (mgr.14164) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:55.637293+0000 mon.smithi047 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.190:0/263905613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:57.017 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:57.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:34:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:57.112426+0000 mgr.smithi047.zgysuz (mgr.14164) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:34:58.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.146+0000 7f1632473700 1 -- 172.21.15.190:0/31535799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 msgr2=0x7f162c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:58.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.146+0000 7f1632473700 1 --2- 172.21.15.190:0/31535799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 0x7f162c0fb570 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f1620009fb0 tx=0x7f1620009b30).stop 2023-11-15T04:34:58.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.146+0000 7f1632473700 1 -- 172.21.15.190:0/31535799 shutdown_connections 2023-11-15T04:34:58.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.146+0000 7f1632473700 1 --2- 172.21.15.190:0/31535799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 0x7f162c0fb570 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:58.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.146+0000 7f1632473700 1 -- 172.21.15.190:0/31535799 >> 172.21.15.190:0/31535799 conn(0x7f162c0f0b70 msgr2=0x7f162c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:58.147 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.146+0000 7f1632473700 1 -- 172.21.15.190:0/31535799 shutdown_connections 2023-11-15T04:34:58.148 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.146+0000 7f1632473700 1 -- 172.21.15.190:0/31535799 wait complete. 2023-11-15T04:34:58.149 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.147+0000 7f1632473700 1 Processor -- start 2023-11-15T04:34:58.150 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.147+0000 7f1632473700 1 -- start start 2023-11-15T04:34:58.150 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.147+0000 7f1632473700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 0x7f162c1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:58.150 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.147+0000 7f1632473700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162c108da0 con 0x7f162c060200 2023-11-15T04:34:58.150 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f162bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 0x7f162c1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:58.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f162bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 0x7f162c1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:54742/0 (socket says 172.21.15.190:54742) 2023-11-15T04:34:58.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f162bfff700 1 -- 172.21.15.190:0/1736170709 learned_addr learned my addr 172.21.15.190:0/1736170709 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:34:58.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f162bfff700 1 -- 172.21.15.190:0/1736170709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620005040 con 0x7f162c060200 2023-11-15T04:34:58.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f162bfff700 1 --2- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 0x7f162c1088b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f1620013510 tx=0x7f162000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:58.151 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f1628ff9700 1 -- 172.21.15.190:0/1736170709 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1620009320 con 0x7f162c060200 2023-11-15T04:34:58.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162c108fa0 con 0x7f162c060200 2023-11-15T04:34:58.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f1628ff9700 1 -- 172.21.15.190:0/1736170709 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1620013f00 con 0x7f162c060200 2023-11-15T04:34:58.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f1628ff9700 1 -- 172.21.15.190:0/1736170709 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1620014dc0 con 0x7f162c060200 2023-11-15T04:34:58.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.148+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162c109430 con 0x7f162c060200 2023-11-15T04:34:58.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.149+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f162c045ad0 con 0x7f162c060200 2023-11-15T04:34:58.152 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.149+0000 7f1628ff9700 1 -- 172.21.15.190:0/1736170709 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1620014420 con 0x7f162c060200 2023-11-15T04:34:58.153 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.149+0000 7f1628ff9700 1 --2- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f161402ff60 0x7f1614032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:34:58.153 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.149+0000 7f1628ff9700 1 -- 172.21.15.190:0/1736170709 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1620041110 con 0x7f162c060200 2023-11-15T04:34:58.153 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.149+0000 7f162b7fe700 1 --2- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f161402ff60 0x7f1614032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:34:58.153 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.150+0000 7f162b7fe700 1 --2- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f161402ff60 0x7f1614032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f161c0096a0 tx=0x7f161c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:34:58.154 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.154+0000 7f1628ff9700 1 -- 172.21.15.190:0/1736170709 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16200095f0 con 0x7f162c060200 2023-11-15T04:34:58.361 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.361+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f162c10a2e0 con 0x7f162c060200 2023-11-15T04:34:58.362 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.362+0000 7f1628ff9700 1 -- 172.21.15.190:0/1736170709 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f162001cb10 con 0x7f162c060200 2023-11-15T04:34:58.362 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:34:58.363 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:34:58.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f161402ff60 msgr2=0x7f1614032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:58.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 --2- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f161402ff60 0x7f1614032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f161c0096a0 tx=0x7f161c006b40).stop 2023-11-15T04:34:58.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 msgr2=0x7f162c1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:34:58.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 --2- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 0x7f162c1088b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f1620013510 tx=0x7f162000b040).stop 2023-11-15T04:34:58.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 shutdown_connections 2023-11-15T04:34:58.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 --2- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f161402ff60 0x7f1614032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:58.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 --2- 172.21.15.190:0/1736170709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f162c060200 0x7f162c1088b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:34:58.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 >> 172.21.15.190:0/1736170709 conn(0x7f162c0f0b70 msgr2=0x7f162c0f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:34:58.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 shutdown_connections 2023-11-15T04:34:58.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:34:58.364+0000 7f1632473700 1 -- 172.21.15.190:0/1736170709 wait complete. 2023-11-15T04:34:58.366 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:34:59.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:58 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:34:58.362782+0000 mon.smithi047 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.190:0/1736170709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:34:59.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:34:59.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:00.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:34:59 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:34:59.112756+0000 mgr.smithi047.zgysuz (mgr.14164) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:00.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.894+0000 7f61e634e700 1 -- 172.21.15.190:0/1128203527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 msgr2=0x7f61e00fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:00.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.894+0000 7f61e634e700 1 --2- 172.21.15.190:0/1128203527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 0x7f61e00fb4f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f61c8009fb0 tx=0x7f61c8009b30).stop 2023-11-15T04:35:00.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.894+0000 7f61e634e700 1 -- 172.21.15.190:0/1128203527 shutdown_connections 2023-11-15T04:35:00.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.894+0000 7f61e634e700 1 --2- 172.21.15.190:0/1128203527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 0x7f61e00fb4f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:00.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.894+0000 7f61e634e700 1 -- 172.21.15.190:0/1128203527 >> 172.21.15.190:0/1128203527 conn(0x7f61e00f0b70 msgr2=0x7f61e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:00.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.894+0000 7f61e634e700 1 -- 172.21.15.190:0/1128203527 shutdown_connections 2023-11-15T04:35:00.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.894+0000 7f61e634e700 1 -- 172.21.15.190:0/1128203527 wait complete. 2023-11-15T04:35:00.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.895+0000 7f61e634e700 1 Processor -- start 2023-11-15T04:35:00.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.895+0000 7f61e634e700 1 -- start start 2023-11-15T04:35:00.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.895+0000 7f61e634e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 0x7f61e01088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:00.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.895+0000 7f61e634e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0108dd0 con 0x7f61e0060280 2023-11-15T04:35:00.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.895+0000 7f61dffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 0x7f61e01088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:00.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.896+0000 7f61dffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 0x7f61e01088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56442/0 (socket says 172.21.15.190:56442) 2023-11-15T04:35:00.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.896+0000 7f61dffff700 1 -- 172.21.15.190:0/448285762 learned_addr learned my addr 172.21.15.190:0/448285762 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:00.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.896+0000 7f61dffff700 1 -- 172.21.15.190:0/448285762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c8005040 con 0x7f61e0060280 2023-11-15T04:35:00.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.896+0000 7f61dffff700 1 --2- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 0x7f61e01088e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f61c8013510 tx=0x7f61c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:00.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.896+0000 7f61dcff9700 1 -- 172.21.15.190:0/448285762 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c8009320 con 0x7f61e0060280 2023-11-15T04:35:00.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.896+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e0108fd0 con 0x7f61e0060280 2023-11-15T04:35:00.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.896+0000 7f61dcff9700 1 -- 172.21.15.190:0/448285762 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61c8013f00 con 0x7f61e0060280 2023-11-15T04:35:00.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.896+0000 7f61dcff9700 1 -- 172.21.15.190:0/448285762 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c8014dc0 con 0x7f61e0060280 2023-11-15T04:35:00.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.896+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e0109460 con 0x7f61e0060280 2023-11-15T04:35:00.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.897+0000 7f61dcff9700 1 -- 172.21.15.190:0/448285762 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61c80095f0 con 0x7f61e0060280 2023-11-15T04:35:00.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.897+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e0045ad0 con 0x7f61e0060280 2023-11-15T04:35:00.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.897+0000 7f61dcff9700 1 --2- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61cc02ff60 0x7f61cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:00.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.897+0000 7f61dcff9700 1 -- 172.21.15.190:0/448285762 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61c8041100 con 0x7f61e0060280 2023-11-15T04:35:00.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.897+0000 7f61df7fe700 1 --2- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61cc02ff60 0x7f61cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:00.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.898+0000 7f61df7fe700 1 --2- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61cc02ff60 0x7f61cc032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f61d00096a0 tx=0x7f61d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:00.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:00.902+0000 7f61dcff9700 1 -- 172.21.15.190:0/448285762 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61c8014420 con 0x7f61e0060280 2023-11-15T04:35:01.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.105+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61e010a310 con 0x7f61e0060280 2023-11-15T04:35:01.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.105+0000 7f61dcff9700 1 -- 172.21.15.190:0/448285762 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f61c8010230 con 0x7f61e0060280 2023-11-15T04:35:01.105 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:01.106 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:01.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61cc02ff60 msgr2=0x7f61cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:01.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 --2- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61cc02ff60 0x7f61cc032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f61d00096a0 tx=0x7f61d0006b40).stop 2023-11-15T04:35:01.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 msgr2=0x7f61e01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:01.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 --2- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 0x7f61e01088e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f61c8013510 tx=0x7f61c800b040).stop 2023-11-15T04:35:01.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 shutdown_connections 2023-11-15T04:35:01.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 --2- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f61cc02ff60 0x7f61cc032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:01.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 --2- 172.21.15.190:0/448285762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e0060280 0x7f61e01088e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:01.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 >> 172.21.15.190:0/448285762 conn(0x7f61e00f0b70 msgr2=0x7f61e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:01.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 shutdown_connections 2023-11-15T04:35:01.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:01.107+0000 7f61e634e700 1 -- 172.21.15.190:0/448285762 wait complete. 2023-11-15T04:35:01.109 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:01.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:01 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:01.105814+0000 mon.smithi047 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.190:0/448285762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:02.488 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:02.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:01.113131+0000 mgr.smithi047.zgysuz (mgr.14164) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:03.620 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.618+0000 7f38ccdf5700 1 -- 172.21.15.190:0/1082839254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 msgr2=0x7f38c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:03.620 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.618+0000 7f38ccdf5700 1 --2- 172.21.15.190:0/1082839254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 0x7f38c80f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f38b8009fb0 tx=0x7f38b8009b30).stop 2023-11-15T04:35:03.620 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.618+0000 7f38ccdf5700 1 -- 172.21.15.190:0/1082839254 shutdown_connections 2023-11-15T04:35:03.621 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.618+0000 7f38ccdf5700 1 --2- 172.21.15.190:0/1082839254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 0x7f38c80f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:03.621 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.618+0000 7f38ccdf5700 1 -- 172.21.15.190:0/1082839254 >> 172.21.15.190:0/1082839254 conn(0x7f38c80f0b70 msgr2=0x7f38c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:03.621 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.619+0000 7f38ccdf5700 1 -- 172.21.15.190:0/1082839254 shutdown_connections 2023-11-15T04:35:03.621 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.619+0000 7f38ccdf5700 1 -- 172.21.15.190:0/1082839254 wait complete. 2023-11-15T04:35:03.621 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.619+0000 7f38ccdf5700 1 Processor -- start 2023-11-15T04:35:03.621 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.619+0000 7f38ccdf5700 1 -- start start 2023-11-15T04:35:03.622 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.619+0000 7f38ccdf5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 0x7f38c8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:03.622 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.619+0000 7f38ccdf5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c8060220 con 0x7f38c80f5410 2023-11-15T04:35:03.622 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.619+0000 7f38c659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 0x7f38c8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:03.622 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.620+0000 7f38c659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 0x7f38c8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56452/0 (socket says 172.21.15.190:56452) 2023-11-15T04:35:03.623 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.620+0000 7f38c659c700 1 -- 172.21.15.190:0/3491964782 learned_addr learned my addr 172.21.15.190:0/3491964782 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:03.623 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.620+0000 7f38c659c700 1 -- 172.21.15.190:0/3491964782 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b8005040 con 0x7f38c80f5410 2023-11-15T04:35:03.623 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.620+0000 7f38c659c700 1 --2- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 0x7f38c8109450 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f38b8013510 tx=0x7f38b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:03.623 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.620+0000 7f38beffd700 1 -- 172.21.15.190:0/3491964782 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38b8009320 con 0x7f38c80f5410 2023-11-15T04:35:03.624 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.620+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c8060420 con 0x7f38c80f5410 2023-11-15T04:35:03.624 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.620+0000 7f38beffd700 1 -- 172.21.15.190:0/3491964782 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38b8013f00 con 0x7f38c80f5410 2023-11-15T04:35:03.624 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.620+0000 7f38beffd700 1 -- 172.21.15.190:0/3491964782 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38b8014d90 con 0x7f38c80f5410 2023-11-15T04:35:03.625 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.620+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c8109b00 con 0x7f38c80f5410 2023-11-15T04:35:03.626 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.621+0000 7f38beffd700 1 -- 172.21.15.190:0/3491964782 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38b80095f0 con 0x7f38c80f5410 2023-11-15T04:35:03.626 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.621+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c8059100 con 0x7f38c80f5410 2023-11-15T04:35:03.626 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.621+0000 7f38beffd700 1 --2- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38b402ff60 0x7f38b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:03.627 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.621+0000 7f38beffd700 1 -- 172.21.15.190:0/3491964782 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38b8041000 con 0x7f38c80f5410 2023-11-15T04:35:03.627 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.621+0000 7f38c5d9b700 1 --2- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38b402ff60 0x7f38b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:03.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.622+0000 7f38c5d9b700 1 --2- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38b402ff60 0x7f38b4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f38b00096a0 tx=0x7f38b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:03.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.626+0000 7f38beffd700 1 -- 172.21.15.190:0/3491964782 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38b8014420 con 0x7f38c80f5410 2023-11-15T04:35:03.833 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.833+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38c8045ad0 con 0x7f38c80f5410 2023-11-15T04:35:03.834 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.834+0000 7f38beffd700 1 -- 172.21.15.190:0/3491964782 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38b801d7f0 con 0x7f38c80f5410 2023-11-15T04:35:03.834 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:03.835 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:03.836 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38b402ff60 msgr2=0x7f38b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:03.836 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 --2- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38b402ff60 0x7f38b4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f38b00096a0 tx=0x7f38b0006b40).stop 2023-11-15T04:35:03.836 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 msgr2=0x7f38c8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:03.837 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 --2- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 0x7f38c8109450 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f38b8013510 tx=0x7f38b800b040).stop 2023-11-15T04:35:03.837 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 shutdown_connections 2023-11-15T04:35:03.837 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 --2- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f38b402ff60 0x7f38b4032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:03.837 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 --2- 172.21.15.190:0/3491964782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80f5410 0x7f38c8109450 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:03.838 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 >> 172.21.15.190:0/3491964782 conn(0x7f38c80f0b70 msgr2=0x7f38c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:03.838 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 shutdown_connections 2023-11-15T04:35:03.839 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:03.836+0000 7f38ccdf5700 1 -- 172.21.15.190:0/3491964782 wait complete. 2023-11-15T04:35:03.839 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:03.113372+0000 mgr.smithi047.zgysuz (mgr.14164) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:03.834580+0000 mon.smithi047 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.190:0/3491964782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:05.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:05.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:06.316 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.314+0000 7f32a469b700 1 -- 172.21.15.190:0/1690681868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 msgr2=0x7f329c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:06.316 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.314+0000 7f32a469b700 1 --2- 172.21.15.190:0/1690681868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 0x7f329c0f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f328c009fb0 tx=0x7f328c009b30).stop 2023-11-15T04:35:06.316 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.314+0000 7f32a469b700 1 -- 172.21.15.190:0/1690681868 shutdown_connections 2023-11-15T04:35:06.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.314+0000 7f32a469b700 1 --2- 172.21.15.190:0/1690681868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 0x7f329c0f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:06.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.314+0000 7f32a469b700 1 -- 172.21.15.190:0/1690681868 >> 172.21.15.190:0/1690681868 conn(0x7f329c0f0b70 msgr2=0x7f329c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:06.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.314+0000 7f32a469b700 1 -- 172.21.15.190:0/1690681868 shutdown_connections 2023-11-15T04:35:06.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.314+0000 7f32a469b700 1 -- 172.21.15.190:0/1690681868 wait complete. 2023-11-15T04:35:06.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.315+0000 7f32a469b700 1 Processor -- start 2023-11-15T04:35:06.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.315+0000 7f32a469b700 1 -- start start 2023-11-15T04:35:06.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.315+0000 7f32a469b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 0x7f329c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:06.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.315+0000 7f32a469b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c060220 con 0x7f329c0f5410 2023-11-15T04:35:06.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.315+0000 7f32a2437700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 0x7f329c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:06.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.315+0000 7f32a2437700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 0x7f329c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56474/0 (socket says 172.21.15.190:56474) 2023-11-15T04:35:06.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.315+0000 7f32a2437700 1 -- 172.21.15.190:0/2336956123 learned_addr learned my addr 172.21.15.190:0/2336956123 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:06.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.316+0000 7f32a2437700 1 -- 172.21.15.190:0/2336956123 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c005040 con 0x7f329c0f5410 2023-11-15T04:35:06.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.316+0000 7f32a2437700 1 --2- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 0x7f329c109450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f328c014000 tx=0x7f328c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:06.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.316+0000 7f3292ffd700 1 -- 172.21.15.190:0/2336956123 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f328c0095f0 con 0x7f329c0f5410 2023-11-15T04:35:06.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.316+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329c060420 con 0x7f329c0f5410 2023-11-15T04:35:06.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.316+0000 7f3292ffd700 1 -- 172.21.15.190:0/2336956123 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f328c007e70 con 0x7f329c0f5410 2023-11-15T04:35:06.320 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.316+0000 7f3292ffd700 1 -- 172.21.15.190:0/2336956123 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f328c014ea0 con 0x7f329c0f5410 2023-11-15T04:35:06.320 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.316+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f329c109b00 con 0x7f329c0f5410 2023-11-15T04:35:06.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.317+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329c059100 con 0x7f329c0f5410 2023-11-15T04:35:06.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.317+0000 7f3292ffd700 1 -- 172.21.15.190:0/2336956123 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f328c025070 con 0x7f329c0f5410 2023-11-15T04:35:06.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.317+0000 7f3292ffd700 1 --2- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f328802ff60 0x7f3288032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:06.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.317+0000 7f3292ffd700 1 -- 172.21.15.190:0/2336956123 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f328c040a00 con 0x7f329c0f5410 2023-11-15T04:35:06.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.317+0000 7f32a1c36700 1 --2- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f328802ff60 0x7f3288032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:06.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.317+0000 7f32a1c36700 1 --2- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f328802ff60 0x7f3288032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f32980096a0 tx=0x7f3298006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:06.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.322+0000 7f3292ffd700 1 -- 172.21.15.190:0/2336956123 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f328c0079c0 con 0x7f329c0f5410 2023-11-15T04:35:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:05.113586+0000 mgr.smithi047.zgysuz (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:05.219796+0000 mon.smithi047 (mon.0) 257 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:35:06.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:05.221361+0000 mon.smithi047 (mon.0) 258 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:35:06.529 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.529+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f329c045ad0 con 0x7f329c0f5410 2023-11-15T04:35:06.530 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.530+0000 7f3292ffd700 1 -- 172.21.15.190:0/2336956123 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f328c00f300 con 0x7f329c0f5410 2023-11-15T04:35:06.531 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:06.531 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:06.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f328802ff60 msgr2=0x7f3288032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:06.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 --2- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f328802ff60 0x7f3288032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f32980096a0 tx=0x7f3298006b40).stop 2023-11-15T04:35:06.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 msgr2=0x7f329c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:06.533 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 --2- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 0x7f329c109450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f328c014000 tx=0x7f328c00b040).stop 2023-11-15T04:35:06.533 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 shutdown_connections 2023-11-15T04:35:06.533 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 --2- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f328802ff60 0x7f3288032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:06.533 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 --2- 172.21.15.190:0/2336956123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329c0f5410 0x7f329c109450 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:06.533 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 >> 172.21.15.190:0/2336956123 conn(0x7f329c0f0b70 msgr2=0x7f329c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:06.534 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 shutdown_connections 2023-11-15T04:35:06.534 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:06.532+0000 7f32a469b700 1 -- 172.21.15.190:0/2336956123 wait complete. 2023-11-15T04:35:06.534 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:07.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:06.530858+0000 mon.smithi047 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.190:0/2336956123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:07.905 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:07.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:08.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:07.113886+0000 mgr.smithi047.zgysuz (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:09.027 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.027+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3820666851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 msgr2=0x7fb4b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:09.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.027+0000 7fb4b9b97700 1 --2- 172.21.15.190:0/3820666851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 0x7fb4b40f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fb4a4009fb0 tx=0x7fb4a4009b30).stop 2023-11-15T04:35:09.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.027+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3820666851 shutdown_connections 2023-11-15T04:35:09.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.027+0000 7fb4b9b97700 1 --2- 172.21.15.190:0/3820666851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 0x7fb4b40f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:09.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.027+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3820666851 >> 172.21.15.190:0/3820666851 conn(0x7fb4b40f0b70 msgr2=0x7fb4b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:09.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.027+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3820666851 shutdown_connections 2023-11-15T04:35:09.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.027+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3820666851 wait complete. 2023-11-15T04:35:09.029 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.028+0000 7fb4b9b97700 1 Processor -- start 2023-11-15T04:35:09.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.028+0000 7fb4b9b97700 1 -- start start 2023-11-15T04:35:09.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.028+0000 7fb4b9b97700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 0x7fb4b4109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:09.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.028+0000 7fb4b9b97700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b4060240 con 0x7fb4b40f5410 2023-11-15T04:35:09.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.028+0000 7fb4b37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 0x7fb4b4109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:09.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.028+0000 7fb4b37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 0x7fb4b4109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56496/0 (socket says 172.21.15.190:56496) 2023-11-15T04:35:09.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.028+0000 7fb4b37fe700 1 -- 172.21.15.190:0/3614047571 learned_addr learned my addr 172.21.15.190:0/3614047571 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:09.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.029+0000 7fb4b37fe700 1 -- 172.21.15.190:0/3614047571 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a4005040 con 0x7fb4b40f5410 2023-11-15T04:35:09.033 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.029+0000 7fb4b37fe700 1 --2- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 0x7fb4b4109400 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fb4a4013510 tx=0x7fb4a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:09.033 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.029+0000 7fb4abfff700 1 -- 172.21.15.190:0/3614047571 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4a4009320 con 0x7fb4b40f5410 2023-11-15T04:35:09.033 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.029+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b4060440 con 0x7fb4b40f5410 2023-11-15T04:35:09.033 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.029+0000 7fb4abfff700 1 -- 172.21.15.190:0/3614047571 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4a4013f00 con 0x7fb4b40f5410 2023-11-15T04:35:09.033 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.029+0000 7fb4abfff700 1 -- 172.21.15.190:0/3614047571 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4a4014d90 con 0x7fb4b40f5410 2023-11-15T04:35:09.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.029+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b4109ab0 con 0x7fb4b40f5410 2023-11-15T04:35:09.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.030+0000 7fb4abfff700 1 -- 172.21.15.190:0/3614047571 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4a40095f0 con 0x7fb4b40f5410 2023-11-15T04:35:09.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.030+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b4045ad0 con 0x7fb4b40f5410 2023-11-15T04:35:09.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.030+0000 7fb4abfff700 1 --2- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4a002ff60 0x7fb4a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:09.034 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.030+0000 7fb4abfff700 1 -- 172.21.15.190:0/3614047571 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4a4041000 con 0x7fb4b40f5410 2023-11-15T04:35:09.035 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.030+0000 7fb4b2ffd700 1 --2- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4a002ff60 0x7fb4a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:09.035 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.031+0000 7fb4b2ffd700 1 --2- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4a002ff60 0x7fb4a0032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb49c0096a0 tx=0x7fb49c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:09.035 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.035+0000 7fb4abfff700 1 -- 172.21.15.190:0/3614047571 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4a4014420 con 0x7fb4b40f5410 2023-11-15T04:35:09.240 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.240+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4b40fae60 con 0x7fb4b40f5410 2023-11-15T04:35:09.241 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.241+0000 7fb4abfff700 1 -- 172.21.15.190:0/3614047571 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4a401d7f0 con 0x7fb4b40f5410 2023-11-15T04:35:09.242 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:09.242 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:09.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4a002ff60 msgr2=0x7fb4a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:09.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 --2- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4a002ff60 0x7fb4a0032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb49c0096a0 tx=0x7fb49c006b40).stop 2023-11-15T04:35:09.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 msgr2=0x7fb4b4109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:09.243 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 --2- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 0x7fb4b4109400 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fb4a4013510 tx=0x7fb4a400b040).stop 2023-11-15T04:35:09.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 shutdown_connections 2023-11-15T04:35:09.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 --2- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb4a002ff60 0x7fb4a0032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:09.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 --2- 172.21.15.190:0/3614047571 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b40f5410 0x7fb4b4109400 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:09.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 >> 172.21.15.190:0/3614047571 conn(0x7fb4b40f0b70 msgr2=0x7fb4b40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:09.244 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 shutdown_connections 2023-11-15T04:35:09.245 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:09.243+0000 7fb4b9b97700 1 -- 172.21.15.190:0/3614047571 wait complete. 2023-11-15T04:35:09.245 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:10.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:09.114241+0000 mgr.smithi047.zgysuz (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:10.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:09.241719+0000 mon.smithi047 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.190:0/3614047571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:10.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:10.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:11.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.763+0000 7f7632e43700 1 -- 172.21.15.190:0/3681820941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 msgr2=0x7f762c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:11.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.763+0000 7f7632e43700 1 --2- 172.21.15.190:0/3681820941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 0x7f762c0f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f7620009fb0 tx=0x7f7620009b30).stop 2023-11-15T04:35:11.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.763+0000 7f7632e43700 1 -- 172.21.15.190:0/3681820941 shutdown_connections 2023-11-15T04:35:11.766 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.763+0000 7f7632e43700 1 --2- 172.21.15.190:0/3681820941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 0x7f762c0f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:11.766 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.763+0000 7f7632e43700 1 -- 172.21.15.190:0/3681820941 >> 172.21.15.190:0/3681820941 conn(0x7f762c0f0b70 msgr2=0x7f762c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:11.766 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.763+0000 7f7632e43700 1 -- 172.21.15.190:0/3681820941 shutdown_connections 2023-11-15T04:35:11.766 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.763+0000 7f7632e43700 1 -- 172.21.15.190:0/3681820941 wait complete. 2023-11-15T04:35:11.766 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.764+0000 7f7632e43700 1 Processor -- start 2023-11-15T04:35:11.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.764+0000 7f7632e43700 1 -- start start 2023-11-15T04:35:11.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.764+0000 7f7632e43700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 0x7f762c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:11.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.764+0000 7f7632e43700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c060220 con 0x7f762c0f5410 2023-11-15T04:35:11.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.764+0000 7f7630bdf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 0x7f762c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:11.767 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.764+0000 7f7630bdf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 0x7f762c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:47996/0 (socket says 172.21.15.190:47996) 2023-11-15T04:35:11.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.764+0000 7f7630bdf700 1 -- 172.21.15.190:0/1740000584 learned_addr learned my addr 172.21.15.190:0/1740000584 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:11.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.765+0000 7f7630bdf700 1 -- 172.21.15.190:0/1740000584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7620005040 con 0x7f762c0f5410 2023-11-15T04:35:11.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.765+0000 7f7630bdf700 1 --2- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 0x7f762c1093f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7620014000 tx=0x7f762000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:11.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.765+0000 7f76297fa700 1 -- 172.21.15.190:0/1740000584 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76200095f0 con 0x7f762c0f5410 2023-11-15T04:35:11.768 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.765+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c060420 con 0x7f762c0f5410 2023-11-15T04:35:11.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.765+0000 7f76297fa700 1 -- 172.21.15.190:0/1740000584 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7620007e70 con 0x7f762c0f5410 2023-11-15T04:35:11.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.765+0000 7f76297fa700 1 -- 172.21.15.190:0/1740000584 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7620014ea0 con 0x7f762c0f5410 2023-11-15T04:35:11.769 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.765+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c109aa0 con 0x7f762c0f5410 2023-11-15T04:35:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.766+0000 7f76297fa700 1 -- 172.21.15.190:0/1740000584 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76200079c0 con 0x7f762c0f5410 2023-11-15T04:35:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.766+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762c059100 con 0x7f762c0f5410 2023-11-15T04:35:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.766+0000 7f76297fa700 1 --2- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f761c02ff60 0x7f761c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:11.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.766+0000 7f76297fa700 1 -- 172.21.15.190:0/1740000584 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7620040920 con 0x7f762c0f5410 2023-11-15T04:35:11.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.766+0000 7f762bfff700 1 --2- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f761c02ff60 0x7f761c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:11.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.767+0000 7f762bfff700 1 --2- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f761c02ff60 0x7f761c032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f76180096a0 tx=0x7f7618006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:11.772 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.771+0000 7f76297fa700 1 -- 172.21.15.190:0/1740000584 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f762000f950 con 0x7f762c0f5410 2023-11-15T04:35:11.977 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.977+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f762c045ad0 con 0x7f762c0f5410 2023-11-15T04:35:11.978 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.978+0000 7f76297fa700 1 -- 172.21.15.190:0/1740000584 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7620010230 con 0x7f762c0f5410 2023-11-15T04:35:11.978 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:11.978 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:11.979 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f761c02ff60 msgr2=0x7f761c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:11.979 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 --2- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f761c02ff60 0x7f761c032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f76180096a0 tx=0x7f7618006b40).stop 2023-11-15T04:35:11.980 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 msgr2=0x7f762c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:11.980 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 --2- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 0x7f762c1093f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7620014000 tx=0x7f762000b040).stop 2023-11-15T04:35:11.980 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 shutdown_connections 2023-11-15T04:35:11.980 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 --2- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f761c02ff60 0x7f761c032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:11.980 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 --2- 172.21.15.190:0/1740000584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f762c0f5410 0x7f762c1093f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:11.981 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 >> 172.21.15.190:0/1740000584 conn(0x7f762c0f0b70 msgr2=0x7f762c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:11.981 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 shutdown_connections 2023-11-15T04:35:11.981 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:11.979+0000 7f7632e43700 1 -- 172.21.15.190:0/1740000584 wait complete. 2023-11-15T04:35:11.981 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:12.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:12 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:11.114610+0000 mgr.smithi047.zgysuz (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:12.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:12 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:11.978265+0000 mon.smithi047 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.190:0/1740000584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:13.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:13.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:13.115066+0000 mgr.smithi047.zgysuz (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:14.533 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.530+0000 7fb58aae9700 1 -- 172.21.15.190:0/3998217163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 msgr2=0x7fb5840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:14.534 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.530+0000 7fb58aae9700 1 --2- 172.21.15.190:0/3998217163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 0x7fb5840f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb578009fb0 tx=0x7fb578009b30).stop 2023-11-15T04:35:14.534 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.530+0000 7fb58aae9700 1 -- 172.21.15.190:0/3998217163 shutdown_connections 2023-11-15T04:35:14.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.530+0000 7fb58aae9700 1 --2- 172.21.15.190:0/3998217163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 0x7fb5840f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:14.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.530+0000 7fb58aae9700 1 -- 172.21.15.190:0/3998217163 >> 172.21.15.190:0/3998217163 conn(0x7fb5840f0b70 msgr2=0x7fb5840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:14.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.530+0000 7fb58aae9700 1 -- 172.21.15.190:0/3998217163 shutdown_connections 2023-11-15T04:35:14.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.530+0000 7fb58aae9700 1 -- 172.21.15.190:0/3998217163 wait complete. 2023-11-15T04:35:14.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.530+0000 7fb58aae9700 1 Processor -- start 2023-11-15T04:35:14.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.530+0000 7fb58aae9700 1 -- start start 2023-11-15T04:35:14.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.531+0000 7fb58aae9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 0x7fb584109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:14.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.531+0000 7fb58aae9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb584060240 con 0x7fb5840f5410 2023-11-15T04:35:14.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.531+0000 7fb588885700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 0x7fb584109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:14.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.531+0000 7fb588885700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 0x7fb584109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:48016/0 (socket says 172.21.15.190:48016) 2023-11-15T04:35:14.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.531+0000 7fb588885700 1 -- 172.21.15.190:0/2835208521 learned_addr learned my addr 172.21.15.190:0/2835208521 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:14.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.531+0000 7fb588885700 1 -- 172.21.15.190:0/2835208521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb578005040 con 0x7fb5840f5410 2023-11-15T04:35:14.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.532+0000 7fb588885700 1 --2- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 0x7fb584109430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb578013510 tx=0x7fb57800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:14.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.532+0000 7fb5817fa700 1 -- 172.21.15.190:0/2835208521 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb578009320 con 0x7fb5840f5410 2023-11-15T04:35:14.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.532+0000 7fb5817fa700 1 -- 172.21.15.190:0/2835208521 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb578013f00 con 0x7fb5840f5410 2023-11-15T04:35:14.543 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.532+0000 7fb5817fa700 1 -- 172.21.15.190:0/2835208521 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb578014d90 con 0x7fb5840f5410 2023-11-15T04:35:14.543 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.532+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb584060440 con 0x7fb5840f5410 2023-11-15T04:35:14.543 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.532+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb584109ae0 con 0x7fb5840f5410 2023-11-15T04:35:14.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.533+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb584045ad0 con 0x7fb5840f5410 2023-11-15T04:35:14.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.533+0000 7fb5817fa700 1 -- 172.21.15.190:0/2835208521 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5780095f0 con 0x7fb5840f5410 2023-11-15T04:35:14.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.533+0000 7fb5817fa700 1 --2- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb57402ff60 0x7fb574032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:14.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.533+0000 7fb5817fa700 1 -- 172.21.15.190:0/2835208521 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb578041000 con 0x7fb5840f5410 2023-11-15T04:35:14.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.533+0000 7fb583fff700 1 --2- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb57402ff60 0x7fb574032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:14.546 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.533+0000 7fb583fff700 1 --2- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb57402ff60 0x7fb574032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fb5700096a0 tx=0x7fb570006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:14.547 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.538+0000 7fb5817fa700 1 -- 172.21.15.190:0/2835208521 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb57801d7f0 con 0x7fb5840f5410 2023-11-15T04:35:14.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.749+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5840fae60 con 0x7fb5840f5410 2023-11-15T04:35:14.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.750+0000 7fb5817fa700 1 -- 172.21.15.190:0/2835208521 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb578010230 con 0x7fb5840f5410 2023-11-15T04:35:14.750 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:14.750 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:14.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.751+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb57402ff60 msgr2=0x7fb574032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:14.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.751+0000 7fb58aae9700 1 --2- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb57402ff60 0x7fb574032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fb5700096a0 tx=0x7fb570006b40).stop 2023-11-15T04:35:14.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.751+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 msgr2=0x7fb584109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:14.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.751+0000 7fb58aae9700 1 --2- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 0x7fb584109430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb578013510 tx=0x7fb57800b040).stop 2023-11-15T04:35:14.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.751+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 shutdown_connections 2023-11-15T04:35:14.753 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.751+0000 7fb58aae9700 1 --2- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb57402ff60 0x7fb574032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:14.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.751+0000 7fb58aae9700 1 --2- 172.21.15.190:0/2835208521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5840f5410 0x7fb584109430 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:14.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.751+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 >> 172.21.15.190:0/2835208521 conn(0x7fb5840f0b70 msgr2=0x7fb5840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:14.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.751+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 shutdown_connections 2023-11-15T04:35:14.754 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:14.752+0000 7fb58aae9700 1 -- 172.21.15.190:0/2835208521 wait complete. 2023-11-15T04:35:14.755 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:15.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:15 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:14.750136+0000 mon.smithi047 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.190:0/2835208521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:16.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:16.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:15.115421+0000 mgr.smithi047.zgysuz (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:17.300 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.299+0000 7f62e95d2700 1 -- 172.21.15.190:0/2075827126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 msgr2=0x7f62e40f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:17.300 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 --2- 172.21.15.190:0/2075827126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 0x7f62e40f55e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f62d4009fb0 tx=0x7f62d4009b30).stop 2023-11-15T04:35:17.300 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 -- 172.21.15.190:0/2075827126 shutdown_connections 2023-11-15T04:35:17.300 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 --2- 172.21.15.190:0/2075827126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 0x7f62e40f55e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:17.301 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 -- 172.21.15.190:0/2075827126 >> 172.21.15.190:0/2075827126 conn(0x7f62e40f0920 msgr2=0x7f62e40f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:17.301 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 -- 172.21.15.190:0/2075827126 shutdown_connections 2023-11-15T04:35:17.301 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 -- 172.21.15.190:0/2075827126 wait complete. 2023-11-15T04:35:17.303 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 Processor -- start 2023-11-15T04:35:17.303 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 -- start start 2023-11-15T04:35:17.303 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 0x7f62e4106590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:17.303 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.300+0000 7f62e95d2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e4106a80 con 0x7f62e40f51c0 2023-11-15T04:35:17.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62e2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 0x7f62e4106590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:17.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62e2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 0x7f62e4106590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:48026/0 (socket says 172.21.15.190:48026) 2023-11-15T04:35:17.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62e2ffd700 1 -- 172.21.15.190:0/2352791395 learned_addr learned my addr 172.21.15.190:0/2352791395 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:17.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62e2ffd700 1 -- 172.21.15.190:0/2352791395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4005040 con 0x7f62e40f51c0 2023-11-15T04:35:17.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62e2ffd700 1 --2- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 0x7f62e4106590 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f62d4013510 tx=0x7f62d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:17.305 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62cb7fe700 1 -- 172.21.15.190:0/2352791395 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d4009320 con 0x7f62e40f51c0 2023-11-15T04:35:17.305 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e4106c80 con 0x7f62e40f51c0 2023-11-15T04:35:17.305 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62cb7fe700 1 -- 172.21.15.190:0/2352791395 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62d4013f00 con 0x7f62e40f51c0 2023-11-15T04:35:17.305 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62cb7fe700 1 -- 172.21.15.190:0/2352791395 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d4014d90 con 0x7f62e40f51c0 2023-11-15T04:35:17.305 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.301+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e4107110 con 0x7f62e40f51c0 2023-11-15T04:35:17.306 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.302+0000 7f62cb7fe700 1 -- 172.21.15.190:0/2352791395 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62d40095f0 con 0x7f62e40f51c0 2023-11-15T04:35:17.306 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.302+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e4045ad0 con 0x7f62e40f51c0 2023-11-15T04:35:17.306 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.302+0000 7f62cb7fe700 1 --2- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f62cc02ff60 0x7f62cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:17.306 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.303+0000 7f62cb7fe700 1 -- 172.21.15.190:0/2352791395 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62d4041000 con 0x7f62e40f51c0 2023-11-15T04:35:17.306 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.303+0000 7f62e27fc700 1 --2- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f62cc02ff60 0x7f62cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:17.307 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.303+0000 7f62e27fc700 1 --2- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f62cc02ff60 0x7f62cc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f62d80096a0 tx=0x7f62d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:17.307 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.307+0000 7f62cb7fe700 1 -- 172.21.15.190:0/2352791395 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62d4014420 con 0x7f62e40f51c0 2023-11-15T04:35:17.514 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.514+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62e4107f10 con 0x7f62e40f51c0 2023-11-15T04:35:17.514 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.514+0000 7f62cb7fe700 1 -- 172.21.15.190:0/2352791395 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62d4010230 con 0x7f62e40f51c0 2023-11-15T04:35:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:17.515 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:17.516 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f62cc02ff60 msgr2=0x7f62cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:17.516 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 --2- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f62cc02ff60 0x7f62cc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f62d80096a0 tx=0x7f62d8006b40).stop 2023-11-15T04:35:17.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 msgr2=0x7f62e4106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:17.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 --2- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 0x7f62e4106590 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f62d4013510 tx=0x7f62d400b040).stop 2023-11-15T04:35:17.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 shutdown_connections 2023-11-15T04:35:17.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 --2- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f62cc02ff60 0x7f62cc032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:17.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 --2- 172.21.15.190:0/2352791395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e40f51c0 0x7f62e4106590 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:17.517 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 >> 172.21.15.190:0/2352791395 conn(0x7f62e40f0920 msgr2=0x7f62e40f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:17.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 shutdown_connections 2023-11-15T04:35:17.518 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:17.516+0000 7f62e95d2700 1 -- 172.21.15.190:0/2352791395 wait complete. 2023-11-15T04:35:17.518 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:18.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:17.115685+0000 mgr.smithi047.zgysuz (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:18.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:17.515045+0000 mon.smithi047 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.190:0/2352791395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:18.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:18.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:20.038 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.036+0000 7f918008b700 1 -- 172.21.15.190:0/648716564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 msgr2=0x7f91780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:20.039 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.036+0000 7f918008b700 1 --2- 172.21.15.190:0/648716564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 0x7f91780f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f9168009fb0 tx=0x7f9168009b30).stop 2023-11-15T04:35:20.039 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.036+0000 7f918008b700 1 -- 172.21.15.190:0/648716564 shutdown_connections 2023-11-15T04:35:20.039 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.036+0000 7f918008b700 1 --2- 172.21.15.190:0/648716564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 0x7f91780f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:20.039 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.036+0000 7f918008b700 1 -- 172.21.15.190:0/648716564 >> 172.21.15.190:0/648716564 conn(0x7f91780f0b70 msgr2=0x7f91780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:20.040 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.036+0000 7f918008b700 1 -- 172.21.15.190:0/648716564 shutdown_connections 2023-11-15T04:35:20.040 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.036+0000 7f918008b700 1 -- 172.21.15.190:0/648716564 wait complete. 2023-11-15T04:35:20.040 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.037+0000 7f918008b700 1 Processor -- start 2023-11-15T04:35:20.040 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.037+0000 7f918008b700 1 -- start start 2023-11-15T04:35:20.040 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.037+0000 7f918008b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 0x7f9178106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:20.041 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.037+0000 7f918008b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9178106c10 con 0x7f91780f5410 2023-11-15T04:35:20.041 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.037+0000 7f917de27700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 0x7f9178106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:20.041 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.037+0000 7f917de27700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 0x7f9178106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:48044/0 (socket says 172.21.15.190:48044) 2023-11-15T04:35:20.041 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.037+0000 7f917de27700 1 -- 172.21.15.190:0/2151689464 learned_addr learned my addr 172.21.15.190:0/2151689464 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:20.041 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.037+0000 7f917de27700 1 -- 172.21.15.190:0/2151689464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168005040 con 0x7f91780f5410 2023-11-15T04:35:20.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.037+0000 7f917de27700 1 --2- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 0x7f9178106720 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f9168013510 tx=0x7f916800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:20.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.038+0000 7f916e7fc700 1 -- 172.21.15.190:0/2151689464 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9168009320 con 0x7f91780f5410 2023-11-15T04:35:20.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.038+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9178106e10 con 0x7f91780f5410 2023-11-15T04:35:20.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.038+0000 7f916e7fc700 1 -- 172.21.15.190:0/2151689464 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9168013f00 con 0x7f91780f5410 2023-11-15T04:35:20.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.038+0000 7f916e7fc700 1 -- 172.21.15.190:0/2151689464 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9168014d90 con 0x7f91780f5410 2023-11-15T04:35:20.043 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.038+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91781072a0 con 0x7f91780f5410 2023-11-15T04:35:20.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.038+0000 7f916e7fc700 1 -- 172.21.15.190:0/2151689464 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9168014420 con 0x7f91780f5410 2023-11-15T04:35:20.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.038+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9178045ad0 con 0x7f91780f5410 2023-11-15T04:35:20.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.039+0000 7f916e7fc700 1 --2- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f916402ff60 0x7f9164032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:20.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.039+0000 7f916e7fc700 1 -- 172.21.15.190:0/2151689464 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9168041050 con 0x7f91780f5410 2023-11-15T04:35:20.044 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.039+0000 7f917d626700 1 --2- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f916402ff60 0x7f9164032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:20.045 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.039+0000 7f917d626700 1 --2- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f916402ff60 0x7f9164032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f91740096a0 tx=0x7f9174006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:20.045 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.043+0000 7f916e7fc700 1 -- 172.21.15.190:0/2151689464 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91680095f0 con 0x7f91780f5410 2023-11-15T04:35:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.250+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91781080f0 con 0x7f91780f5410 2023-11-15T04:35:20.252 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.251+0000 7f916e7fc700 1 -- 172.21.15.190:0/2151689464 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f916801db70 con 0x7f91780f5410 2023-11-15T04:35:20.252 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:20.252 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:20.253 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.252+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f916402ff60 msgr2=0x7f9164032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:20.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.252+0000 7f918008b700 1 --2- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f916402ff60 0x7f9164032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f91740096a0 tx=0x7f9174006b40).stop 2023-11-15T04:35:20.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.252+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 msgr2=0x7f9178106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:20.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.252+0000 7f918008b700 1 --2- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 0x7f9178106720 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f9168013510 tx=0x7f916800b040).stop 2023-11-15T04:35:20.254 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.252+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 shutdown_connections 2023-11-15T04:35:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.252+0000 7f918008b700 1 --2- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f916402ff60 0x7f9164032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.252+0000 7f918008b700 1 --2- 172.21.15.190:0/2151689464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91780f5410 0x7f9178106720 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.253+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 >> 172.21.15.190:0/2151689464 conn(0x7f91780f0b70 msgr2=0x7f917805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.253+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 shutdown_connections 2023-11-15T04:35:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:20.253+0000 7f918008b700 1 -- 172.21.15.190:0/2151689464 wait complete. 2023-11-15T04:35:20.256 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:20.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:20 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:19.116060+0000 mgr.smithi047.zgysuz (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:21.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:21 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:20.252409+0000 mon.smithi047 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.190:0/2151689464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:21.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:21.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:21.116470+0000 mgr.smithi047.zgysuz (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:22.824 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.822+0000 7f915a594700 1 -- 172.21.15.190:0/4149750185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 msgr2=0x7f91540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:22.824 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.822+0000 7f915a594700 1 --2- 172.21.15.190:0/4149750185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 0x7f91540f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f913c009fb0 tx=0x7f913c009b30).stop 2023-11-15T04:35:22.824 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.822+0000 7f915a594700 1 -- 172.21.15.190:0/4149750185 shutdown_connections 2023-11-15T04:35:22.824 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.822+0000 7f915a594700 1 --2- 172.21.15.190:0/4149750185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 0x7f91540f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:22.825 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.822+0000 7f915a594700 1 -- 172.21.15.190:0/4149750185 >> 172.21.15.190:0/4149750185 conn(0x7f91540f0b70 msgr2=0x7f91540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:22.825 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.822+0000 7f915a594700 1 -- 172.21.15.190:0/4149750185 shutdown_connections 2023-11-15T04:35:22.825 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.822+0000 7f915a594700 1 -- 172.21.15.190:0/4149750185 wait complete. 2023-11-15T04:35:22.825 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.823+0000 7f915a594700 1 Processor -- start 2023-11-15T04:35:22.825 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.823+0000 7f915a594700 1 -- start start 2023-11-15T04:35:22.825 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.823+0000 7f915a594700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 0x7f915410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:22.826 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.823+0000 7f915a594700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9154107240 con 0x7f91540f5410 2023-11-15T04:35:22.827 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.823+0000 7f9153fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 0x7f915410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:22.827 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.823+0000 7f9153fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 0x7f915410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:33990/0 (socket says 172.21.15.190:33990) 2023-11-15T04:35:22.828 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.823+0000 7f9153fff700 1 -- 172.21.15.190:0/497666254 learned_addr learned my addr 172.21.15.190:0/497666254 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:22.828 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.824+0000 7f9153fff700 1 -- 172.21.15.190:0/497666254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c005040 con 0x7f91540f5410 2023-11-15T04:35:22.828 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.824+0000 7f9153fff700 1 --2- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 0x7f915410b850 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f913c013510 tx=0x7f913c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:22.828 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.824+0000 7f9150ff9700 1 -- 172.21.15.190:0/497666254 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f913c009320 con 0x7f91540f5410 2023-11-15T04:35:22.828 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.824+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9154060220 con 0x7f91540f5410 2023-11-15T04:35:22.829 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.824+0000 7f9150ff9700 1 -- 172.21.15.190:0/497666254 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f913c013f00 con 0x7f91540f5410 2023-11-15T04:35:22.829 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.824+0000 7f9150ff9700 1 -- 172.21.15.190:0/497666254 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f913c014d90 con 0x7f91540f5410 2023-11-15T04:35:22.829 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.824+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915410bd40 con 0x7f91540f5410 2023-11-15T04:35:22.829 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.825+0000 7f9150ff9700 1 -- 172.21.15.190:0/497666254 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f913c0095f0 con 0x7f91540f5410 2023-11-15T04:35:22.829 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.825+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9154045ad0 con 0x7f91540f5410 2023-11-15T04:35:22.829 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.825+0000 7f9150ff9700 1 --2- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f914002ff60 0x7f9140032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:22.830 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.825+0000 7f9150ff9700 1 -- 172.21.15.190:0/497666254 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f913c040fe0 con 0x7f91540f5410 2023-11-15T04:35:22.830 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.825+0000 7f91537fe700 1 --2- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f914002ff60 0x7f9140032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:22.830 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.826+0000 7f91537fe700 1 --2- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f914002ff60 0x7f9140032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f91440096a0 tx=0x7f9144006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:22.831 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:22.830+0000 7f9150ff9700 1 -- 172.21.15.190:0/497666254 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f913c01d7f0 con 0x7f91540f5410 2023-11-15T04:35:23.038 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.038+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91540fae60 con 0x7f91540f5410 2023-11-15T04:35:23.039 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.038+0000 7f9150ff9700 1 -- 172.21.15.190:0/497666254 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f913c010230 con 0x7f91540f5410 2023-11-15T04:35:23.040 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:23.040 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:23.041 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f914002ff60 msgr2=0x7f9140032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:23.041 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 --2- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f914002ff60 0x7f9140032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f91440096a0 tx=0x7f9144006b40).stop 2023-11-15T04:35:23.041 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 msgr2=0x7f915410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:23.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 --2- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 0x7f915410b850 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f913c013510 tx=0x7f913c00b040).stop 2023-11-15T04:35:23.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 shutdown_connections 2023-11-15T04:35:23.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 --2- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f914002ff60 0x7f9140032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:23.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 --2- 172.21.15.190:0/497666254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91540f5410 0x7f915410b850 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:23.042 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 >> 172.21.15.190:0/497666254 conn(0x7f91540f0b70 msgr2=0x7f91540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:23.043 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 shutdown_connections 2023-11-15T04:35:23.043 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:23.040+0000 7f915a594700 1 -- 172.21.15.190:0/497666254 wait complete. 2023-11-15T04:35:23.043 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:23.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:23 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:23.040038+0000 mon.smithi047 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.190:0/497666254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:24.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:24.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:24.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:23.116833+0000 mgr.smithi047.zgysuz (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:25.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.586+0000 7f39d65e1700 1 -- 172.21.15.190:0/2559495088 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 msgr2=0x7f39d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:25.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.586+0000 7f39d65e1700 1 --2- 172.21.15.190:0/2559495088 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 0x7f39d00f55c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f39cc009fb0 tx=0x7f39cc009b30).stop 2023-11-15T04:35:25.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.586+0000 7f39d65e1700 1 -- 172.21.15.190:0/2559495088 shutdown_connections 2023-11-15T04:35:25.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.586+0000 7f39d65e1700 1 --2- 172.21.15.190:0/2559495088 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 0x7f39d00f55c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:25.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.586+0000 7f39d65e1700 1 -- 172.21.15.190:0/2559495088 >> 172.21.15.190:0/2559495088 conn(0x7f39d00f09a0 msgr2=0x7f39d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:25.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.586+0000 7f39d65e1700 1 -- 172.21.15.190:0/2559495088 shutdown_connections 2023-11-15T04:35:25.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.586+0000 7f39d65e1700 1 -- 172.21.15.190:0/2559495088 wait complete. 2023-11-15T04:35:25.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.586+0000 7f39d65e1700 1 Processor -- start 2023-11-15T04:35:25.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.586+0000 7f39d65e1700 1 -- start start 2023-11-15T04:35:25.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.587+0000 7f39d65e1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 0x7f39d01064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:25.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.587+0000 7f39d65e1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d01069b0 con 0x7f39d00f51a0 2023-11-15T04:35:25.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.587+0000 7f39d55df700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 0x7f39d01064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:25.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.587+0000 7f39d55df700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 0x7f39d01064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:33998/0 (socket says 172.21.15.190:33998) 2023-11-15T04:35:25.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.587+0000 7f39d55df700 1 -- 172.21.15.190:0/2916344983 learned_addr learned my addr 172.21.15.190:0/2916344983 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:25.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.587+0000 7f39d55df700 1 -- 172.21.15.190:0/2916344983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc005040 con 0x7f39d00f51a0 2023-11-15T04:35:25.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.587+0000 7f39d55df700 1 --2- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 0x7f39d01064c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f39cc013510 tx=0x7f39cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:25.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.588+0000 7f39c5ffb700 1 -- 172.21.15.190:0/2916344983 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39cc009320 con 0x7f39d00f51a0 2023-11-15T04:35:25.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.588+0000 7f39c5ffb700 1 -- 172.21.15.190:0/2916344983 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39cc013f00 con 0x7f39d00f51a0 2023-11-15T04:35:25.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.588+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d0106bb0 con 0x7f39d00f51a0 2023-11-15T04:35:25.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.588+0000 7f39c5ffb700 1 -- 172.21.15.190:0/2916344983 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39cc014dc0 con 0x7f39d00f51a0 2023-11-15T04:35:25.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.588+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d0107040 con 0x7f39d00f51a0 2023-11-15T04:35:25.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.589+0000 7f39c5ffb700 1 -- 172.21.15.190:0/2916344983 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39cc0095f0 con 0x7f39d00f51a0 2023-11-15T04:35:25.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.589+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d0045ad0 con 0x7f39d00f51a0 2023-11-15T04:35:25.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.589+0000 7f39c5ffb700 1 --2- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f39bc02ff60 0x7f39bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:25.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.589+0000 7f39c5ffb700 1 -- 172.21.15.190:0/2916344983 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39cc041100 con 0x7f39d00f51a0 2023-11-15T04:35:25.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.589+0000 7f39d4dde700 1 --2- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f39bc02ff60 0x7f39bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:25.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.589+0000 7f39d4dde700 1 --2- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f39bc02ff60 0x7f39bc032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f39c00096a0 tx=0x7f39c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:25.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.594+0000 7f39c5ffb700 1 -- 172.21.15.190:0/2916344983 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39cc014420 con 0x7f39d00f51a0 2023-11-15T04:35:25.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.801+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39d0107ef0 con 0x7f39d00f51a0 2023-11-15T04:35:25.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.801+0000 7f39c5ffb700 1 -- 172.21.15.190:0/2916344983 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f39cc010230 con 0x7f39d00f51a0 2023-11-15T04:35:25.803 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:25.803 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:25.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f39bc02ff60 msgr2=0x7f39bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:25.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 --2- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f39bc02ff60 0x7f39bc032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f39c00096a0 tx=0x7f39c0006b40).stop 2023-11-15T04:35:25.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 msgr2=0x7f39d01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:25.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 --2- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 0x7f39d01064c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f39cc013510 tx=0x7f39cc00b040).stop 2023-11-15T04:35:25.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 shutdown_connections 2023-11-15T04:35:25.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 --2- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f39bc02ff60 0x7f39bc032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:25.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 --2- 172.21.15.190:0/2916344983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39d00f51a0 0x7f39d01064c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:25.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 >> 172.21.15.190:0/2916344983 conn(0x7f39d00f09a0 msgr2=0x7f39d00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:25.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 shutdown_connections 2023-11-15T04:35:25.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:25.803+0000 7f39d65e1700 1 -- 172.21.15.190:0/2916344983 wait complete. 2023-11-15T04:35:25.807 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:26.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:25.117227+0000 mgr.smithi047.zgysuz (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:26.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:25.802779+0000 mon.smithi047 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.190:0/2916344983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:27.190 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:27.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:28.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:27.117584+0000 mgr.smithi047.zgysuz (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:28.328 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.326+0000 7fcf3080d700 1 -- 172.21.15.190:0/4113907826 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 msgr2=0x7fcf2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:28.328 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.326+0000 7fcf3080d700 1 --2- 172.21.15.190:0/4113907826 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 0x7fcf2c0f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fcf20009fb0 tx=0x7fcf20009b30).stop 2023-11-15T04:35:28.328 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.326+0000 7fcf3080d700 1 -- 172.21.15.190:0/4113907826 shutdown_connections 2023-11-15T04:35:28.328 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.326+0000 7fcf3080d700 1 --2- 172.21.15.190:0/4113907826 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 0x7fcf2c0f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:28.329 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.326+0000 7fcf3080d700 1 -- 172.21.15.190:0/4113907826 >> 172.21.15.190:0/4113907826 conn(0x7fcf2c0f0b70 msgr2=0x7fcf2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:28.329 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.326+0000 7fcf3080d700 1 -- 172.21.15.190:0/4113907826 shutdown_connections 2023-11-15T04:35:28.329 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.326+0000 7fcf3080d700 1 -- 172.21.15.190:0/4113907826 wait complete. 2023-11-15T04:35:28.329 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.327+0000 7fcf3080d700 1 Processor -- start 2023-11-15T04:35:28.329 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.327+0000 7fcf3080d700 1 -- start start 2023-11-15T04:35:28.330 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.327+0000 7fcf3080d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 0x7fcf2c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:28.330 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.327+0000 7fcf3080d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c060220 con 0x7fcf2c0f5410 2023-11-15T04:35:28.330 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.327+0000 7fcf2a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 0x7fcf2c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:28.330 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.327+0000 7fcf2a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 0x7fcf2c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:34030/0 (socket says 172.21.15.190:34030) 2023-11-15T04:35:28.330 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.327+0000 7fcf2a59c700 1 -- 172.21.15.190:0/168755122 learned_addr learned my addr 172.21.15.190:0/168755122 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:28.332 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.328+0000 7fcf2a59c700 1 -- 172.21.15.190:0/168755122 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf20005040 con 0x7fcf2c0f5410 2023-11-15T04:35:28.332 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.328+0000 7fcf2a59c700 1 --2- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 0x7fcf2c109450 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fcf20014000 tx=0x7fcf2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:28.332 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.328+0000 7fcf1affd700 1 -- 172.21.15.190:0/168755122 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf200095f0 con 0x7fcf2c0f5410 2023-11-15T04:35:28.332 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.328+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c060420 con 0x7fcf2c0f5410 2023-11-15T04:35:28.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.328+0000 7fcf1affd700 1 -- 172.21.15.190:0/168755122 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf20007e70 con 0x7fcf2c0f5410 2023-11-15T04:35:28.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.328+0000 7fcf1affd700 1 -- 172.21.15.190:0/168755122 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf20014ea0 con 0x7fcf2c0f5410 2023-11-15T04:35:28.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.328+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c109b00 con 0x7fcf2c0f5410 2023-11-15T04:35:28.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.329+0000 7fcf1affd700 1 -- 172.21.15.190:0/168755122 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf20025070 con 0x7fcf2c0f5410 2023-11-15T04:35:28.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.329+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c059100 con 0x7fcf2c0f5410 2023-11-15T04:35:28.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.329+0000 7fcf1affd700 1 --2- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcf1402ff60 0x7fcf14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:28.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.329+0000 7fcf1affd700 1 -- 172.21.15.190:0/168755122 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf20040920 con 0x7fcf2c0f5410 2023-11-15T04:35:28.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.329+0000 7fcf29d9b700 1 --2- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcf1402ff60 0x7fcf14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:28.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.329+0000 7fcf29d9b700 1 --2- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcf1402ff60 0x7fcf14032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fcf1c0096a0 tx=0x7fcf1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:28.335 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.334+0000 7fcf1affd700 1 -- 172.21.15.190:0/168755122 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf2000f4e0 con 0x7fcf2c0f5410 2023-11-15T04:35:28.544 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.544+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf2c045ad0 con 0x7fcf2c0f5410 2023-11-15T04:35:28.545 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.544+0000 7fcf1affd700 1 -- 172.21.15.190:0/168755122 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf2000f300 con 0x7fcf2c0f5410 2023-11-15T04:35:28.546 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:28.546 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:28.547 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcf1402ff60 msgr2=0x7fcf14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:28.547 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 --2- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcf1402ff60 0x7fcf14032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fcf1c0096a0 tx=0x7fcf1c006b40).stop 2023-11-15T04:35:28.547 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 msgr2=0x7fcf2c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:28.548 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 --2- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 0x7fcf2c109450 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fcf20014000 tx=0x7fcf2000b040).stop 2023-11-15T04:35:28.548 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 shutdown_connections 2023-11-15T04:35:28.548 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 --2- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fcf1402ff60 0x7fcf14032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:28.548 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 --2- 172.21.15.190:0/168755122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf2c0f5410 0x7fcf2c109450 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:28.548 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 >> 172.21.15.190:0/168755122 conn(0x7fcf2c0f0b70 msgr2=0x7fcf2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:28.548 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 shutdown_connections 2023-11-15T04:35:28.549 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:28.546+0000 7fcf3080d700 1 -- 172.21.15.190:0/168755122 wait complete. 2023-11-15T04:35:28.549 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:29.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:28 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:28.545953+0000 mon.smithi047 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.190:0/168755122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:29.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:29.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:30.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:29 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:29.117962+0000 mgr.smithi047.zgysuz (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:31.086 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 -- 172.21.15.190:0/605894208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 msgr2=0x7f34f80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:31.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 --2- 172.21.15.190:0/605894208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 0x7f34f80f55c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f34e8009fb0 tx=0x7f34e8009b30).stop 2023-11-15T04:35:31.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 -- 172.21.15.190:0/605894208 shutdown_connections 2023-11-15T04:35:31.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 --2- 172.21.15.190:0/605894208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 0x7f34f80f55c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:31.087 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 -- 172.21.15.190:0/605894208 >> 172.21.15.190:0/605894208 conn(0x7f34f80f09a0 msgr2=0x7f34f80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:31.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 -- 172.21.15.190:0/605894208 shutdown_connections 2023-11-15T04:35:31.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 -- 172.21.15.190:0/605894208 wait complete. 2023-11-15T04:35:31.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 Processor -- start 2023-11-15T04:35:31.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 -- start start 2023-11-15T04:35:31.088 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 0x7f34f81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:31.089 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.084+0000 7f34fe3c8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f8060220 con 0x7f34f80f51a0 2023-11-15T04:35:31.089 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.085+0000 7f34fd3c6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 0x7f34f81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:31.089 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.085+0000 7f34fd3c6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 0x7f34f81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:35624/0 (socket says 172.21.15.190:35624) 2023-11-15T04:35:31.089 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.085+0000 7f34fd3c6700 1 -- 172.21.15.190:0/1590713799 learned_addr learned my addr 172.21.15.190:0/1590713799 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:31.090 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.085+0000 7f34fd3c6700 1 -- 172.21.15.190:0/1590713799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e8005040 con 0x7f34f80f51a0 2023-11-15T04:35:31.090 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.085+0000 7f34fd3c6700 1 --2- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 0x7f34f81091d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f34e8013510 tx=0x7f34e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:31.090 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.085+0000 7f34edffb700 1 -- 172.21.15.190:0/1590713799 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e8009320 con 0x7f34f80f51a0 2023-11-15T04:35:31.090 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.085+0000 7f34edffb700 1 -- 172.21.15.190:0/1590713799 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34e8013f00 con 0x7f34f80f51a0 2023-11-15T04:35:31.091 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.085+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f8060420 con 0x7f34f80f51a0 2023-11-15T04:35:31.091 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.085+0000 7f34edffb700 1 -- 172.21.15.190:0/1590713799 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e8014dc0 con 0x7f34f80f51a0 2023-11-15T04:35:31.091 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.086+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f8109880 con 0x7f34f80f51a0 2023-11-15T04:35:31.092 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.086+0000 7f34edffb700 1 -- 172.21.15.190:0/1590713799 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34e801d3d0 con 0x7f34f80f51a0 2023-11-15T04:35:31.092 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.087+0000 7f34edffb700 1 --2- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f34e402ff60 0x7f34e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:31.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.087+0000 7f34edffb700 1 -- 172.21.15.190:0/1590713799 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34e8041050 con 0x7f34f80f51a0 2023-11-15T04:35:31.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.087+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f8059100 con 0x7f34f80f51a0 2023-11-15T04:35:31.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.087+0000 7f34fcbc5700 1 --2- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f34e402ff60 0x7f34e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:31.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.087+0000 7f34fcbc5700 1 --2- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f34e402ff60 0x7f34e4032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f34f40096a0 tx=0x7f34f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:31.093 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.091+0000 7f34edffb700 1 -- 172.21.15.190:0/1590713799 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34e8012920 con 0x7f34f80f51a0 2023-11-15T04:35:31.300 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.299+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34f80fabe0 con 0x7f34f80f51a0 2023-11-15T04:35:31.301 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.300+0000 7f34edffb700 1 -- 172.21.15.190:0/1590713799 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f34e802b020 con 0x7f34f80f51a0 2023-11-15T04:35:31.301 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:31.302 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:31.303 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f34e402ff60 msgr2=0x7f34e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:31.303 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 --2- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f34e402ff60 0x7f34e4032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f34f40096a0 tx=0x7f34f4006b40).stop 2023-11-15T04:35:31.303 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 msgr2=0x7f34f81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:31.303 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 --2- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 0x7f34f81091d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f34e8013510 tx=0x7f34e800b040).stop 2023-11-15T04:35:31.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 shutdown_connections 2023-11-15T04:35:31.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 --2- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f34e402ff60 0x7f34e4032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:31.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 --2- 172.21.15.190:0/1590713799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34f80f51a0 0x7f34f81091d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:31.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 >> 172.21.15.190:0/1590713799 conn(0x7f34f80f09a0 msgr2=0x7f34f80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:31.304 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 shutdown_connections 2023-11-15T04:35:31.305 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:31.302+0000 7f34fe3c8700 1 -- 172.21.15.190:0/1590713799 wait complete. 2023-11-15T04:35:31.305 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:31.118269+0000 mgr.smithi047.zgysuz (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:31.301601+0000 mon.smithi047 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.190:0/1590713799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:32.676 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:32.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:33.814 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.813+0000 7efe40442700 1 -- 172.21.15.190:0/3257292242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 msgr2=0x7efe380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:33.814 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.813+0000 7efe40442700 1 --2- 172.21.15.190:0/3257292242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 0x7efe380f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7efe28009fb0 tx=0x7efe28009b30).stop 2023-11-15T04:35:33.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.813+0000 7efe40442700 1 -- 172.21.15.190:0/3257292242 shutdown_connections 2023-11-15T04:35:33.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.813+0000 7efe40442700 1 --2- 172.21.15.190:0/3257292242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 0x7efe380f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:33.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.813+0000 7efe40442700 1 -- 172.21.15.190:0/3257292242 >> 172.21.15.190:0/3257292242 conn(0x7efe380f0b70 msgr2=0x7efe380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:33.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.813+0000 7efe40442700 1 -- 172.21.15.190:0/3257292242 shutdown_connections 2023-11-15T04:35:33.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.813+0000 7efe40442700 1 -- 172.21.15.190:0/3257292242 wait complete. 2023-11-15T04:35:33.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.814+0000 7efe40442700 1 Processor -- start 2023-11-15T04:35:33.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.814+0000 7efe40442700 1 -- start start 2023-11-15T04:35:33.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.814+0000 7efe40442700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 0x7efe38109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:33.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.814+0000 7efe40442700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe38060280 con 0x7efe380f5410 2023-11-15T04:35:33.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.814+0000 7efe3e1de700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 0x7efe38109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:33.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.814+0000 7efe3e1de700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 0x7efe38109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:35648/0 (socket says 172.21.15.190:35648) 2023-11-15T04:35:33.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.814+0000 7efe3e1de700 1 -- 172.21.15.190:0/1574054360 learned_addr learned my addr 172.21.15.190:0/1574054360 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:33.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.815+0000 7efe3e1de700 1 -- 172.21.15.190:0/1574054360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe28005040 con 0x7efe380f5410 2023-11-15T04:35:33.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.815+0000 7efe3e1de700 1 --2- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 0x7efe38109390 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7efe28013510 tx=0x7efe2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:33.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.815+0000 7efe2effd700 1 -- 172.21.15.190:0/1574054360 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe28009320 con 0x7efe380f5410 2023-11-15T04:35:33.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.815+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe38060480 con 0x7efe380f5410 2023-11-15T04:35:33.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.815+0000 7efe2effd700 1 -- 172.21.15.190:0/1574054360 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe28013f00 con 0x7efe380f5410 2023-11-15T04:35:33.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.815+0000 7efe2effd700 1 -- 172.21.15.190:0/1574054360 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe28014d90 con 0x7efe380f5410 2023-11-15T04:35:33.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.815+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe38109ac0 con 0x7efe380f5410 2023-11-15T04:35:33.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.816+0000 7efe2effd700 1 -- 172.21.15.190:0/1574054360 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe280263a0 con 0x7efe380f5410 2023-11-15T04:35:33.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.816+0000 7efe2effd700 1 --2- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2402ffb0 0x7efe24032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:33.821 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.816+0000 7efe2effd700 1 -- 172.21.15.190:0/1574054360 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe28041390 con 0x7efe380f5410 2023-11-15T04:35:33.821 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.816+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe38059100 con 0x7efe380f5410 2023-11-15T04:35:33.821 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.816+0000 7efe3d9dd700 1 --2- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2402ffb0 0x7efe24032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:33.821 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.817+0000 7efe3d9dd700 1 --2- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2402ffb0 0x7efe24032470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7efe340096a0 tx=0x7efe34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:33.822 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:33.821+0000 7efe2effd700 1 -- 172.21.15.190:0/1574054360 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe2801cb10 con 0x7efe380f5410 2023-11-15T04:35:34.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.027+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe38045ad0 con 0x7efe380f5410 2023-11-15T04:35:34.028 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.028+0000 7efe2effd700 1 -- 172.21.15.190:0/1574054360 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe2802b030 con 0x7efe380f5410 2023-11-15T04:35:34.029 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:34.029 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:34.030 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2402ffb0 msgr2=0x7efe24032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:34.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 --2- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2402ffb0 0x7efe24032470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7efe340096a0 tx=0x7efe34006b40).stop 2023-11-15T04:35:34.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 msgr2=0x7efe38109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:34.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 --2- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 0x7efe38109390 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7efe28013510 tx=0x7efe2800b040).stop 2023-11-15T04:35:34.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 shutdown_connections 2023-11-15T04:35:34.031 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 --2- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efe2402ffb0 0x7efe24032470 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:34.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 --2- 172.21.15.190:0/1574054360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe380f5410 0x7efe38109390 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:34.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 >> 172.21.15.190:0/1574054360 conn(0x7efe380f0b70 msgr2=0x7efe380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:34.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 shutdown_connections 2023-11-15T04:35:34.032 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:34.029+0000 7efe40442700 1 -- 172.21.15.190:0/1574054360 wait complete. 2023-11-15T04:35:34.032 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:34.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:33.118560+0000 mgr.smithi047.zgysuz (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:34.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:34.029249+0000 mon.smithi047 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.190:0/1574054360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:35.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:35.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:35.118901+0000 mgr.smithi047.zgysuz (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:36.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 -- 172.21.15.190:0/3059701244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 msgr2=0x7fd9540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:36.576 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 --2- 172.21.15.190:0/3059701244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 0x7fd9540f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fd950009fb0 tx=0x7fd950009b30).stop 2023-11-15T04:35:36.576 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 -- 172.21.15.190:0/3059701244 shutdown_connections 2023-11-15T04:35:36.576 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 --2- 172.21.15.190:0/3059701244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 0x7fd9540f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:36.576 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 -- 172.21.15.190:0/3059701244 >> 172.21.15.190:0/3059701244 conn(0x7fd9540f1c70 msgr2=0x7fd9540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:36.577 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 -- 172.21.15.190:0/3059701244 shutdown_connections 2023-11-15T04:35:36.577 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 -- 172.21.15.190:0/3059701244 wait complete. 2023-11-15T04:35:36.577 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 Processor -- start 2023-11-15T04:35:36.577 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 -- start start 2023-11-15T04:35:36.578 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 0x7fd954106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:36.578 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.574+0000 7fd95b998700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd954106c20 con 0x7fd9540f6510 2023-11-15T04:35:36.578 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.575+0000 7fd959734700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 0x7fd954106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:36.578 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.575+0000 7fd959734700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 0x7fd954106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:35660/0 (socket says 172.21.15.190:35660) 2023-11-15T04:35:36.578 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.575+0000 7fd959734700 1 -- 172.21.15.190:0/2204065577 learned_addr learned my addr 172.21.15.190:0/2204065577 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:36.580 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.575+0000 7fd959734700 1 -- 172.21.15.190:0/2204065577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950005040 con 0x7fd9540f6510 2023-11-15T04:35:36.580 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.575+0000 7fd959734700 1 --2- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 0x7fd954106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fd950013510 tx=0x7fd95000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:36.580 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.575+0000 7fd949ffb700 1 -- 172.21.15.190:0/2204065577 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd950009320 con 0x7fd9540f6510 2023-11-15T04:35:36.581 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.575+0000 7fd949ffb700 1 -- 172.21.15.190:0/2204065577 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd950013f00 con 0x7fd9540f6510 2023-11-15T04:35:36.581 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.575+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd954106e20 con 0x7fd9540f6510 2023-11-15T04:35:36.581 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.575+0000 7fd949ffb700 1 -- 172.21.15.190:0/2204065577 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd950014d90 con 0x7fd9540f6510 2023-11-15T04:35:36.582 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.576+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9541072b0 con 0x7fd9540f6510 2023-11-15T04:35:36.582 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.576+0000 7fd949ffb700 1 -- 172.21.15.190:0/2204065577 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9500095f0 con 0x7fd9540f6510 2023-11-15T04:35:36.582 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.577+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd954045ad0 con 0x7fd9540f6510 2023-11-15T04:35:36.582 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.577+0000 7fd949ffb700 1 --2- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd94002ff60 0x7fd940032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:36.582 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.577+0000 7fd949ffb700 1 -- 172.21.15.190:0/2204065577 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd950041040 con 0x7fd9540f6510 2023-11-15T04:35:36.583 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.577+0000 7fd958f33700 1 --2- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd94002ff60 0x7fd940032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:36.583 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.577+0000 7fd958f33700 1 --2- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd94002ff60 0x7fd940032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd9440096a0 tx=0x7fd944006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:36.583 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.581+0000 7fd949ffb700 1 -- 172.21.15.190:0/2204065577 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd950014420 con 0x7fd9540f6510 2023-11-15T04:35:36.788 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.788+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9540fbf60 con 0x7fd9540f6510 2023-11-15T04:35:36.789 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.788+0000 7fd949ffb700 1 -- 172.21.15.190:0/2204065577 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd950010230 con 0x7fd9540f6510 2023-11-15T04:35:36.790 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:36.790 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:36.791 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd94002ff60 msgr2=0x7fd940032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:36.791 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 --2- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd94002ff60 0x7fd940032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd9440096a0 tx=0x7fd944006b40).stop 2023-11-15T04:35:36.791 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 msgr2=0x7fd954106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:36.791 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 --2- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 0x7fd954106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fd950013510 tx=0x7fd95000b040).stop 2023-11-15T04:35:36.792 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 shutdown_connections 2023-11-15T04:35:36.792 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 --2- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fd94002ff60 0x7fd940032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:36.792 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 --2- 172.21.15.190:0/2204065577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9540f6510 0x7fd954106730 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:36.792 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 >> 172.21.15.190:0/2204065577 conn(0x7fd9540f1c70 msgr2=0x7fd9540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:36.792 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 shutdown_connections 2023-11-15T04:35:36.793 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:36.790+0000 7fd95b998700 1 -- 172.21.15.190:0/2204065577 wait complete. 2023-11-15T04:35:36.793 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:38.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:36.789967+0000 mon.smithi047 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.190:0/2204065577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:38.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:37.119228+0000 mgr.smithi047.zgysuz (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:38.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:38.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:39.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 -- 172.21.15.190:0/126319340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 msgr2=0x7f61300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:39.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 --2- 172.21.15.190:0/126319340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 0x7f61300f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f6124009fb0 tx=0x7f6124009b30).stop 2023-11-15T04:35:39.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 -- 172.21.15.190:0/126319340 shutdown_connections 2023-11-15T04:35:39.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 --2- 172.21.15.190:0/126319340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 0x7f61300f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:39.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 -- 172.21.15.190:0/126319340 >> 172.21.15.190:0/126319340 conn(0x7f61300f1c70 msgr2=0x7f61300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:39.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 -- 172.21.15.190:0/126319340 shutdown_connections 2023-11-15T04:35:39.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 -- 172.21.15.190:0/126319340 wait complete. 2023-11-15T04:35:39.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 Processor -- start 2023-11-15T04:35:39.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 -- start start 2023-11-15T04:35:39.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 0x7f613006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:39.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.372+0000 7f6137d46700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6130069e20 con 0x7f61300f6510 2023-11-15T04:35:39.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.373+0000 7f6135ae2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 0x7f613006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:39.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.373+0000 7f6135ae2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 0x7f613006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:35686/0 (socket says 172.21.15.190:35686) 2023-11-15T04:35:39.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.373+0000 7f6135ae2700 1 -- 172.21.15.190:0/2166002236 learned_addr learned my addr 172.21.15.190:0/2166002236 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:39.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.373+0000 7f6135ae2700 1 -- 172.21.15.190:0/2166002236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124005040 con 0x7f61300f6510 2023-11-15T04:35:39.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.373+0000 7f6135ae2700 1 --2- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 0x7f613006b9a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6124013510 tx=0x7f612400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:39.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.374+0000 7f61227fc700 1 -- 172.21.15.190:0/2166002236 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6124009320 con 0x7f61300f6510 2023-11-15T04:35:39.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.374+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6130069fc0 con 0x7f61300f6510 2023-11-15T04:35:39.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.374+0000 7f61227fc700 1 -- 172.21.15.190:0/2166002236 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6124013f00 con 0x7f61300f6510 2023-11-15T04:35:39.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.374+0000 7f61227fc700 1 -- 172.21.15.190:0/2166002236 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6124014d90 con 0x7f61300f6510 2023-11-15T04:35:39.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.374+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613006a450 con 0x7f61300f6510 2023-11-15T04:35:39.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.374+0000 7f61227fc700 1 -- 172.21.15.190:0/2166002236 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61240095f0 con 0x7f61300f6510 2023-11-15T04:35:39.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.375+0000 7f61227fc700 1 --2- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f611c02ff60 0x7f611c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:39.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.375+0000 7f61227fc700 1 -- 172.21.15.190:0/2166002236 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6124041000 con 0x7f61300f6510 2023-11-15T04:35:39.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.375+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6130059100 con 0x7f61300f6510 2023-11-15T04:35:39.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.375+0000 7f61352e1700 1 --2- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f611c02ff60 0x7f611c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:39.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.375+0000 7f61352e1700 1 --2- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f611c02ff60 0x7f611c032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f612c0096a0 tx=0x7f612c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:39.381 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.380+0000 7f61227fc700 1 -- 172.21.15.190:0/2166002236 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6124014420 con 0x7f61300f6510 2023-11-15T04:35:39.586 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.585+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f613006aab0 con 0x7f61300f6510 2023-11-15T04:35:39.587 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.586+0000 7f61227fc700 1 -- 172.21.15.190:0/2166002236 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f612402b020 con 0x7f61300f6510 2023-11-15T04:35:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:39.587 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:39.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.587+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f611c02ff60 msgr2=0x7f611c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:39.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.587+0000 7f6137d46700 1 --2- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f611c02ff60 0x7f611c032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f612c0096a0 tx=0x7f612c006b40).stop 2023-11-15T04:35:39.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.587+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 msgr2=0x7f613006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:39.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.587+0000 7f6137d46700 1 --2- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 0x7f613006b9a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6124013510 tx=0x7f612400b040).stop 2023-11-15T04:35:39.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.587+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 shutdown_connections 2023-11-15T04:35:39.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.587+0000 7f6137d46700 1 --2- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f611c02ff60 0x7f611c032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:39.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.587+0000 7f6137d46700 1 --2- 172.21.15.190:0/2166002236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61300f6510 0x7f613006b9a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:39.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.587+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 >> 172.21.15.190:0/2166002236 conn(0x7f61300f1c70 msgr2=0x7f61300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:39.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.588+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 shutdown_connections 2023-11-15T04:35:39.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:39.588+0000 7f6137d46700 1 -- 172.21.15.190:0/2166002236 wait complete. 2023-11-15T04:35:39.590 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:40.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:39.119583+0000 mgr.smithi047.zgysuz (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:40.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:39.587455+0000 mon.smithi047 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.190:0/2166002236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:40.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:40.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:42.163 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.158+0000 7f69fdb6d700 1 -- 172.21.15.190:0/3096797700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 msgr2=0x7f69f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:42.166 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.158+0000 7f69fdb6d700 1 --2- 172.21.15.190:0/3096797700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 0x7f69f80f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f69e0009fb0 tx=0x7f69e0009b30).stop 2023-11-15T04:35:42.166 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.158+0000 7f69fdb6d700 1 -- 172.21.15.190:0/3096797700 shutdown_connections 2023-11-15T04:35:42.167 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.158+0000 7f69fdb6d700 1 --2- 172.21.15.190:0/3096797700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 0x7f69f80f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:42.167 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.158+0000 7f69fdb6d700 1 -- 172.21.15.190:0/3096797700 >> 172.21.15.190:0/3096797700 conn(0x7f69f80f1c70 msgr2=0x7f69f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:42.167 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.158+0000 7f69fdb6d700 1 -- 172.21.15.190:0/3096797700 shutdown_connections 2023-11-15T04:35:42.168 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.158+0000 7f69fdb6d700 1 -- 172.21.15.190:0/3096797700 wait complete. 2023-11-15T04:35:42.168 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.159+0000 7f69fdb6d700 1 Processor -- start 2023-11-15T04:35:42.168 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.159+0000 7f69fdb6d700 1 -- start start 2023-11-15T04:35:42.168 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.159+0000 7f69fdb6d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 0x7f69f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:42.168 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.159+0000 7f69fdb6d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f8106c20 con 0x7f69f80f6510 2023-11-15T04:35:42.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.159+0000 7f69f77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 0x7f69f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:42.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.159+0000 7f69f77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 0x7f69f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:47420/0 (socket says 172.21.15.190:47420) 2023-11-15T04:35:42.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.159+0000 7f69f77fe700 1 -- 172.21.15.190:0/1319675696 learned_addr learned my addr 172.21.15.190:0/1319675696 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:42.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.160+0000 7f69f77fe700 1 -- 172.21.15.190:0/1319675696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e0005040 con 0x7f69f80f6510 2023-11-15T04:35:42.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.160+0000 7f69f77fe700 1 --2- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 0x7f69f8106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f69e0013510 tx=0x7f69e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:42.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.160+0000 7f69effff700 1 -- 172.21.15.190:0/1319675696 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69e0009320 con 0x7f69f80f6510 2023-11-15T04:35:42.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.160+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f8106e20 con 0x7f69f80f6510 2023-11-15T04:35:42.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.160+0000 7f69effff700 1 -- 172.21.15.190:0/1319675696 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69e0013f00 con 0x7f69f80f6510 2023-11-15T04:35:42.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.160+0000 7f69effff700 1 -- 172.21.15.190:0/1319675696 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69e0014d90 con 0x7f69f80f6510 2023-11-15T04:35:42.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.160+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f81072b0 con 0x7f69f80f6510 2023-11-15T04:35:42.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.161+0000 7f69effff700 1 -- 172.21.15.190:0/1319675696 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69e00095f0 con 0x7f69f80f6510 2023-11-15T04:35:42.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.161+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f8045ad0 con 0x7f69f80f6510 2023-11-15T04:35:42.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.161+0000 7f69effff700 1 --2- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f69e402ff60 0x7f69e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:42.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.161+0000 7f69effff700 1 -- 172.21.15.190:0/1319675696 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69e0041000 con 0x7f69f80f6510 2023-11-15T04:35:42.171 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.161+0000 7f69f6ffd700 1 --2- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f69e402ff60 0x7f69e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:42.172 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.162+0000 7f69f6ffd700 1 --2- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f69e402ff60 0x7f69e4032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f69e80096a0 tx=0x7f69e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:42.172 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.166+0000 7f69effff700 1 -- 172.21.15.190:0/1319675696 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69e0014420 con 0x7f69f80f6510 2023-11-15T04:35:42.372 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.370+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69f80fbf60 con 0x7f69f80f6510 2023-11-15T04:35:42.372 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.371+0000 7f69effff700 1 -- 172.21.15.190:0/1319675696 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f69e0010230 con 0x7f69f80f6510 2023-11-15T04:35:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:42.373 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:42.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.372+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f69e402ff60 msgr2=0x7f69e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:42.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.372+0000 7f69fdb6d700 1 --2- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f69e402ff60 0x7f69e4032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f69e80096a0 tx=0x7f69e8006b40).stop 2023-11-15T04:35:42.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.372+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 msgr2=0x7f69f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:42.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.372+0000 7f69fdb6d700 1 --2- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 0x7f69f8106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f69e0013510 tx=0x7f69e000b040).stop 2023-11-15T04:35:42.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.373+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 shutdown_connections 2023-11-15T04:35:42.375 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.373+0000 7f69fdb6d700 1 --2- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f69e402ff60 0x7f69e4032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:42.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.373+0000 7f69fdb6d700 1 --2- 172.21.15.190:0/1319675696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69f80f6510 0x7f69f8106730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:42.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.373+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 >> 172.21.15.190:0/1319675696 conn(0x7f69f80f1c70 msgr2=0x7f69f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:42.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.373+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 shutdown_connections 2023-11-15T04:35:42.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:42.373+0000 7f69fdb6d700 1 -- 172.21.15.190:0/1319675696 wait complete. 2023-11-15T04:35:42.376 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:42.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:41.119912+0000 mgr.smithi047.zgysuz (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:43.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:43 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:42.372584+0000 mon.smithi047 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.190:0/1319675696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:43.773 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:43.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:43.120323+0000 mgr.smithi047.zgysuz (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:44.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.910+0000 7f7b50d9b700 1 -- 172.21.15.190:0/538898288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 msgr2=0x7f7b4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:44.915 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.910+0000 7f7b50d9b700 1 --2- 172.21.15.190:0/538898288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f7b34009fb0 tx=0x7f7b34009b30).stop 2023-11-15T04:35:44.915 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.910+0000 7f7b50d9b700 1 -- 172.21.15.190:0/538898288 shutdown_connections 2023-11-15T04:35:44.916 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.910+0000 7f7b50d9b700 1 --2- 172.21.15.190:0/538898288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:44.916 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.910+0000 7f7b50d9b700 1 -- 172.21.15.190:0/538898288 >> 172.21.15.190:0/538898288 conn(0x7f7b4c0f0b70 msgr2=0x7f7b4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:44.916 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.910+0000 7f7b50d9b700 1 -- 172.21.15.190:0/538898288 shutdown_connections 2023-11-15T04:35:44.917 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.910+0000 7f7b50d9b700 1 -- 172.21.15.190:0/538898288 wait complete. 2023-11-15T04:35:44.917 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.911+0000 7f7b50d9b700 1 Processor -- start 2023-11-15T04:35:44.917 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.911+0000 7f7b50d9b700 1 -- start start 2023-11-15T04:35:44.918 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.911+0000 7f7b50d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:44.918 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.911+0000 7f7b50d9b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0fb820 con 0x7f7b4c0f5410 2023-11-15T04:35:44.918 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.911+0000 7f7b4a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:44.919 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.911+0000 7f7b4a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:47438/0 (socket says 172.21.15.190:47438) 2023-11-15T04:35:44.919 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.911+0000 7f7b4a59c700 1 -- 172.21.15.190:0/4200729911 learned_addr learned my addr 172.21.15.190:0/4200729911 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:44.919 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.911+0000 7f7b4a59c700 1 -- 172.21.15.190:0/4200729911 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b34005040 con 0x7f7b4c0f5410 2023-11-15T04:35:44.919 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.912+0000 7f7b4a59c700 1 --2- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f7b34013510 tx=0x7f7b3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:44.920 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.912+0000 7f7b42ffd700 1 -- 172.21.15.190:0/4200729911 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b34009320 con 0x7f7b4c0f5410 2023-11-15T04:35:44.920 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.912+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c0fb9c0 con 0x7f7b4c0f5410 2023-11-15T04:35:44.920 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.912+0000 7f7b42ffd700 1 -- 172.21.15.190:0/4200729911 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b34013f00 con 0x7f7b4c0f5410 2023-11-15T04:35:44.920 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.912+0000 7f7b42ffd700 1 -- 172.21.15.190:0/4200729911 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b34014d90 con 0x7f7b4c0f5410 2023-11-15T04:35:44.920 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.912+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c0fbe50 con 0x7f7b4c0f5410 2023-11-15T04:35:44.921 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.913+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c059100 con 0x7f7b4c0f5410 2023-11-15T04:35:44.921 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.913+0000 7f7b42ffd700 1 -- 172.21.15.190:0/4200729911 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b340095f0 con 0x7f7b4c0f5410 2023-11-15T04:35:44.921 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.913+0000 7f7b42ffd700 1 --2- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b3802ff60 0x7f7b38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:44.921 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.913+0000 7f7b42ffd700 1 -- 172.21.15.190:0/4200729911 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b34041000 con 0x7f7b4c0f5410 2023-11-15T04:35:44.921 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.913+0000 7f7b49d9b700 1 --2- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b3802ff60 0x7f7b38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:44.922 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.913+0000 7f7b49d9b700 1 --2- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b3802ff60 0x7f7b38032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f7b3c0096a0 tx=0x7f7b3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:44.922 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:44.918+0000 7f7b42ffd700 1 -- 172.21.15.190:0/4200729911 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b34014420 con 0x7f7b4c0f5410 2023-11-15T04:35:45.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.124+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b4c0fae60 con 0x7f7b4c0f5410 2023-11-15T04:35:45.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.124+0000 7f7b42ffd700 1 -- 172.21.15.190:0/4200729911 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b3401d7f0 con 0x7f7b4c0f5410 2023-11-15T04:35:45.126 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:45.126 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:45.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b3802ff60 msgr2=0x7f7b38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:45.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 --2- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b3802ff60 0x7f7b38032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f7b3c0096a0 tx=0x7f7b3c006b40).stop 2023-11-15T04:35:45.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 msgr2=0x7f7b4c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:45.128 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 --2- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f7b34013510 tx=0x7f7b3400b040).stop 2023-11-15T04:35:45.128 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 shutdown_connections 2023-11-15T04:35:45.128 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 --2- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b3802ff60 0x7f7b38032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:45.128 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 --2- 172.21.15.190:0/4200729911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:45.128 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 >> 172.21.15.190:0/4200729911 conn(0x7f7b4c0f0b70 msgr2=0x7f7b4c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:45.129 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 shutdown_connections 2023-11-15T04:35:45.129 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:45.126+0000 7f7b50d9b700 1 -- 172.21.15.190:0/4200729911 wait complete. 2023-11-15T04:35:45.129 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:45 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:45.126115+0000 mon.smithi047 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.190:0/4200729911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:45.120720+0000 mgr.smithi047.zgysuz (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:46.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:46.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:47.759 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.757+0000 7f848caec700 1 -- 172.21.15.190:0/2938188246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 msgr2=0x7f84880f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:47.759 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.757+0000 7f848caec700 1 --2- 172.21.15.190:0/2938188246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 0x7f84880f9150 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f847c009fb0 tx=0x7f847c009b30).stop 2023-11-15T04:35:47.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.757+0000 7f848caec700 1 -- 172.21.15.190:0/2938188246 shutdown_connections 2023-11-15T04:35:47.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.757+0000 7f848caec700 1 --2- 172.21.15.190:0/2938188246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 0x7f84880f9150 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:47.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.757+0000 7f848caec700 1 -- 172.21.15.190:0/2938188246 >> 172.21.15.190:0/2938188246 conn(0x7f84880f0b70 msgr2=0x7f84880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:47.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.757+0000 7f848caec700 1 -- 172.21.15.190:0/2938188246 shutdown_connections 2023-11-15T04:35:47.760 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.757+0000 7f848caec700 1 -- 172.21.15.190:0/2938188246 wait complete. 2023-11-15T04:35:47.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.758+0000 7f848caec700 1 Processor -- start 2023-11-15T04:35:47.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.758+0000 7f848caec700 1 -- start start 2023-11-15T04:35:47.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.758+0000 7f848caec700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 0x7f84881088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:47.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.758+0000 7f848caec700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488108da0 con 0x7f8488060220 2023-11-15T04:35:47.761 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.758+0000 7f848659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 0x7f84881088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:47.762 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.758+0000 7f848659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 0x7f84881088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:47442/0 (socket says 172.21.15.190:47442) 2023-11-15T04:35:47.762 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.758+0000 7f848659c700 1 -- 172.21.15.190:0/4197062910 learned_addr learned my addr 172.21.15.190:0/4197062910 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:47.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.759+0000 7f848659c700 1 -- 172.21.15.190:0/4197062910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c005040 con 0x7f8488060220 2023-11-15T04:35:47.763 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.759+0000 7f848659c700 1 --2- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 0x7f84881088b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f847c013510 tx=0x7f847c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:47.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.759+0000 7f8476ffd700 1 -- 172.21.15.190:0/4197062910 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f847c009320 con 0x7f8488060220 2023-11-15T04:35:47.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.759+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8488108fa0 con 0x7f8488060220 2023-11-15T04:35:47.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.759+0000 7f8476ffd700 1 -- 172.21.15.190:0/4197062910 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f847c013f00 con 0x7f8488060220 2023-11-15T04:35:47.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.759+0000 7f8476ffd700 1 -- 172.21.15.190:0/4197062910 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f847c014dc0 con 0x7f8488060220 2023-11-15T04:35:47.764 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.759+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8488109430 con 0x7f8488060220 2023-11-15T04:35:47.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.760+0000 7f8476ffd700 1 -- 172.21.15.190:0/4197062910 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f847c0095f0 con 0x7f8488060220 2023-11-15T04:35:47.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.760+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8488045ad0 con 0x7f8488060220 2023-11-15T04:35:47.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.760+0000 7f8476ffd700 1 --2- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f847002ff60 0x7f8470032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:47.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.760+0000 7f8476ffd700 1 -- 172.21.15.190:0/4197062910 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f847c041000 con 0x7f8488060220 2023-11-15T04:35:47.765 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.760+0000 7f8485d9b700 1 --2- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f847002ff60 0x7f8470032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:47.766 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.761+0000 7f8485d9b700 1 --2- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f847002ff60 0x7f8470032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f84780096a0 tx=0x7f8478006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:47.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.765+0000 7f8476ffd700 1 -- 172.21.15.190:0/4197062910 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f847c01c790 con 0x7f8488060220 2023-11-15T04:35:47.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.969+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f848810a2e0 con 0x7f8488060220 2023-11-15T04:35:47.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.969+0000 7f8476ffd700 1 -- 172.21.15.190:0/4197062910 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f847c010230 con 0x7f8488060220 2023-11-15T04:35:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:47.971 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:47.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f847002ff60 msgr2=0x7f8470032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:47.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 --2- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f847002ff60 0x7f8470032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f84780096a0 tx=0x7f8478006b40).stop 2023-11-15T04:35:47.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 msgr2=0x7f84881088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:47.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 --2- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 0x7f84881088b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f847c013510 tx=0x7f847c00b040).stop 2023-11-15T04:35:47.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 shutdown_connections 2023-11-15T04:35:47.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 --2- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f847002ff60 0x7f8470032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:47.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 --2- 172.21.15.190:0/4197062910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8488060220 0x7f84881088b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:47.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 >> 172.21.15.190:0/4197062910 conn(0x7f84880f0b70 msgr2=0x7f84880f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:47.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 shutdown_connections 2023-11-15T04:35:47.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:47.971+0000 7f848caec700 1 -- 172.21.15.190:0/4197062910 wait complete. 2023-11-15T04:35:47.974 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:47.120974+0000 mgr.smithi047.zgysuz (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:49.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:48 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:47.970860+0000 mon.smithi047 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.190:0/4197062910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:49.336 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:49.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:50.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:49 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:49.121196+0000 mgr.smithi047.zgysuz (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:50.503 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.502+0000 7fe7f608d700 1 -- 172.21.15.190:0/4029664764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 msgr2=0x7fe7f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:50.503 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.502+0000 7fe7f608d700 1 --2- 172.21.15.190:0/4029664764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 0x7fe7f00f55c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fe7e0009fb0 tx=0x7fe7e0009b30).stop 2023-11-15T04:35:50.503 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.502+0000 7fe7f608d700 1 -- 172.21.15.190:0/4029664764 shutdown_connections 2023-11-15T04:35:50.504 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.502+0000 7fe7f608d700 1 --2- 172.21.15.190:0/4029664764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 0x7fe7f00f55c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:50.504 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.502+0000 7fe7f608d700 1 -- 172.21.15.190:0/4029664764 >> 172.21.15.190:0/4029664764 conn(0x7fe7f00f09a0 msgr2=0x7fe7f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:50.504 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.502+0000 7fe7f608d700 1 -- 172.21.15.190:0/4029664764 shutdown_connections 2023-11-15T04:35:50.504 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.502+0000 7fe7f608d700 1 -- 172.21.15.190:0/4029664764 wait complete. 2023-11-15T04:35:50.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.503+0000 7fe7f608d700 1 Processor -- start 2023-11-15T04:35:50.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.503+0000 7fe7f608d700 1 -- start start 2023-11-15T04:35:50.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.503+0000 7fe7f608d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 0x7fe7f01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:50.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.503+0000 7fe7f608d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f01058a0 con 0x7fe7f00f51a0 2023-11-15T04:35:50.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.503+0000 7fe7f508b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 0x7fe7f01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:50.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.503+0000 7fe7f508b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 0x7fe7f01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60404/0 (socket says 172.21.15.190:60404) 2023-11-15T04:35:50.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.503+0000 7fe7f508b700 1 -- 172.21.15.190:0/2160610574 learned_addr learned my addr 172.21.15.190:0/2160610574 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:50.510 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.503+0000 7fe7f508b700 1 -- 172.21.15.190:0/2160610574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0005040 con 0x7fe7f00f51a0 2023-11-15T04:35:50.510 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.504+0000 7fe7f508b700 1 --2- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 0x7fe7f01053b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fe7e0013510 tx=0x7fe7e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:50.510 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.504+0000 7fe7e5ffb700 1 -- 172.21.15.190:0/2160610574 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e0009320 con 0x7fe7f00f51a0 2023-11-15T04:35:50.510 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.504+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f0105aa0 con 0x7fe7f00f51a0 2023-11-15T04:35:50.511 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.504+0000 7fe7e5ffb700 1 -- 172.21.15.190:0/2160610574 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7e0013f00 con 0x7fe7f00f51a0 2023-11-15T04:35:50.511 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.504+0000 7fe7e5ffb700 1 -- 172.21.15.190:0/2160610574 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e0014dc0 con 0x7fe7f00f51a0 2023-11-15T04:35:50.511 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.504+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f0105f30 con 0x7fe7f00f51a0 2023-11-15T04:35:50.511 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.505+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f0045ad0 con 0x7fe7f00f51a0 2023-11-15T04:35:50.511 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.505+0000 7fe7e5ffb700 1 -- 172.21.15.190:0/2160610574 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7e00095f0 con 0x7fe7f00f51a0 2023-11-15T04:35:50.512 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.505+0000 7fe7e5ffb700 1 --2- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe7dc02ff60 0x7fe7dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:50.512 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.505+0000 7fe7e5ffb700 1 -- 172.21.15.190:0/2160610574 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7e0041040 con 0x7fe7f00f51a0 2023-11-15T04:35:50.512 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.505+0000 7fe7f488a700 1 --2- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe7dc02ff60 0x7fe7dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:50.512 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.506+0000 7fe7f488a700 1 --2- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe7dc02ff60 0x7fe7dc032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe7ec0096a0 tx=0x7fe7ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:50.513 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.510+0000 7fe7e5ffb700 1 -- 172.21.15.190:0/2160610574 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7e0014420 con 0x7fe7f00f51a0 2023-11-15T04:35:50.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.717+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7f0106de0 con 0x7fe7f00f51a0 2023-11-15T04:35:50.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.718+0000 7fe7e5ffb700 1 -- 172.21.15.190:0/2160610574 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe7e0010230 con 0x7fe7f00f51a0 2023-11-15T04:35:50.719 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:50.719 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:50.720 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe7dc02ff60 msgr2=0x7fe7dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:50.720 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 --2- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe7dc02ff60 0x7fe7dc032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe7ec0096a0 tx=0x7fe7ec006b40).stop 2023-11-15T04:35:50.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 msgr2=0x7fe7f01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:50.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 --2- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 0x7fe7f01053b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fe7e0013510 tx=0x7fe7e000b040).stop 2023-11-15T04:35:50.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 shutdown_connections 2023-11-15T04:35:50.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 --2- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe7dc02ff60 0x7fe7dc032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:50.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 --2- 172.21.15.190:0/2160610574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7f00f51a0 0x7fe7f01053b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:50.722 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 >> 172.21.15.190:0/2160610574 conn(0x7fe7f00f09a0 msgr2=0x7fe7f00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:50.722 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 shutdown_connections 2023-11-15T04:35:50.722 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:50.719+0000 7fe7f608d700 1 -- 172.21.15.190:0/2160610574 wait complete. 2023-11-15T04:35:50.723 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:51.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:50 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:50.719272+0000 mon.smithi047 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.190:0/2160610574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:52.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:51 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:51.121465+0000 mgr.smithi047.zgysuz (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:52.106 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:52.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:53.191 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.189+0000 7f6c44122700 1 -- 172.21.15.190:0/2637257451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 msgr2=0x7f6c3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:53.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.189+0000 7f6c44122700 1 --2- 172.21.15.190:0/2637257451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 0x7f6c3c0f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f6c2c009fb0 tx=0x7f6c2c009b30).stop 2023-11-15T04:35:53.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.189+0000 7f6c44122700 1 -- 172.21.15.190:0/2637257451 shutdown_connections 2023-11-15T04:35:53.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.189+0000 7f6c44122700 1 --2- 172.21.15.190:0/2637257451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 0x7f6c3c0f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:53.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.189+0000 7f6c44122700 1 -- 172.21.15.190:0/2637257451 >> 172.21.15.190:0/2637257451 conn(0x7f6c3c0f0b70 msgr2=0x7f6c3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:53.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.189+0000 7f6c44122700 1 -- 172.21.15.190:0/2637257451 shutdown_connections 2023-11-15T04:35:53.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.189+0000 7f6c44122700 1 -- 172.21.15.190:0/2637257451 wait complete. 2023-11-15T04:35:53.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.189+0000 7f6c44122700 1 Processor -- start 2023-11-15T04:35:53.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.189+0000 7f6c44122700 1 -- start start 2023-11-15T04:35:53.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.190+0000 7f6c44122700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 0x7f6c3c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:53.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.190+0000 7f6c44122700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c060240 con 0x7f6c3c0f5410 2023-11-15T04:35:53.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.190+0000 7f6c41ebe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 0x7f6c3c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:53.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.190+0000 7f6c41ebe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 0x7f6c3c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60418/0 (socket says 172.21.15.190:60418) 2023-11-15T04:35:53.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.190+0000 7f6c41ebe700 1 -- 172.21.15.190:0/945913127 learned_addr learned my addr 172.21.15.190:0/945913127 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:53.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.190+0000 7f6c41ebe700 1 -- 172.21.15.190:0/945913127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c005040 con 0x7f6c3c0f5410 2023-11-15T04:35:53.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.190+0000 7f6c41ebe700 1 --2- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 0x7f6c3c109430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f6c2c013510 tx=0x7f6c2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:53.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.190+0000 7f6c327fc700 1 -- 172.21.15.190:0/945913127 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c2c009320 con 0x7f6c3c0f5410 2023-11-15T04:35:53.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.191+0000 7f6c327fc700 1 -- 172.21.15.190:0/945913127 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c2c013f00 con 0x7f6c3c0f5410 2023-11-15T04:35:53.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.191+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c060440 con 0x7f6c3c0f5410 2023-11-15T04:35:53.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.191+0000 7f6c327fc700 1 -- 172.21.15.190:0/945913127 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c2c014d90 con 0x7f6c3c0f5410 2023-11-15T04:35:53.196 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.191+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c109ae0 con 0x7f6c3c0f5410 2023-11-15T04:35:53.197 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.191+0000 7f6c327fc700 1 -- 172.21.15.190:0/945913127 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c2c0095f0 con 0x7f6c3c0f5410 2023-11-15T04:35:53.197 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.192+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c045ad0 con 0x7f6c3c0f5410 2023-11-15T04:35:53.197 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.192+0000 7f6c327fc700 1 --2- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6c2802ff60 0x7f6c28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:53.197 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.192+0000 7f6c327fc700 1 -- 172.21.15.190:0/945913127 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c2c041000 con 0x7f6c3c0f5410 2023-11-15T04:35:53.197 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.192+0000 7f6c416bd700 1 --2- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6c2802ff60 0x7f6c28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:53.198 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.192+0000 7f6c416bd700 1 --2- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6c2802ff60 0x7f6c28032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f6c380096a0 tx=0x7f6c38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:53.198 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.197+0000 7f6c327fc700 1 -- 172.21.15.190:0/945913127 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c2c014420 con 0x7f6c3c0f5410 2023-11-15T04:35:53.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.403+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c3c0fae60 con 0x7f6c3c0f5410 2023-11-15T04:35:53.405 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.404+0000 7f6c327fc700 1 -- 172.21.15.190:0/945913127 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c2c01d7f0 con 0x7f6c3c0f5410 2023-11-15T04:35:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:53.405 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:53.407 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6c2802ff60 msgr2=0x7f6c28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:53.407 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 --2- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6c2802ff60 0x7f6c28032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f6c380096a0 tx=0x7f6c38006b40).stop 2023-11-15T04:35:53.407 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 msgr2=0x7f6c3c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:53.407 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 --2- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 0x7f6c3c109430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f6c2c013510 tx=0x7f6c2c00b040).stop 2023-11-15T04:35:53.407 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 shutdown_connections 2023-11-15T04:35:53.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 --2- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f6c2802ff60 0x7f6c28032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:53.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 --2- 172.21.15.190:0/945913127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c3c0f5410 0x7f6c3c109430 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:53.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 >> 172.21.15.190:0/945913127 conn(0x7f6c3c0f0b70 msgr2=0x7f6c3c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:53.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 shutdown_connections 2023-11-15T04:35:53.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:53.406+0000 7f6c44122700 1 -- 172.21.15.190:0/945913127 wait complete. 2023-11-15T04:35:53.409 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:53.121833+0000 mgr.smithi047.zgysuz (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:53.405675+0000 mon.smithi047 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.190:0/945913127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:54.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:54.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:55.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.893+0000 7f4560597700 1 -- 172.21.15.190:0/4168331926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 msgr2=0x7f45580f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:55.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.893+0000 7f4560597700 1 --2- 172.21.15.190:0/4168331926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 0x7f45580f5df0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f454c009fb0 tx=0x7f454c009b30).stop 2023-11-15T04:35:55.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.893+0000 7f4560597700 1 -- 172.21.15.190:0/4168331926 shutdown_connections 2023-11-15T04:35:55.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.893+0000 7f4560597700 1 --2- 172.21.15.190:0/4168331926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 0x7f45580f5df0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:55.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.893+0000 7f4560597700 1 -- 172.21.15.190:0/4168331926 >> 172.21.15.190:0/4168331926 conn(0x7f45580f02e0 msgr2=0x7f45580f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:55.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.893+0000 7f4560597700 1 -- 172.21.15.190:0/4168331926 shutdown_connections 2023-11-15T04:35:55.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.893+0000 7f4560597700 1 -- 172.21.15.190:0/4168331926 wait complete. 2023-11-15T04:35:55.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.894+0000 7f4560597700 1 Processor -- start 2023-11-15T04:35:55.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.894+0000 7f4560597700 1 -- start start 2023-11-15T04:35:55.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.894+0000 7f4560597700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 0x7f45580fd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:55.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.894+0000 7f4560597700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45580fb810 con 0x7f45580f59d0 2023-11-15T04:35:55.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.894+0000 7f455e333700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 0x7f45580fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:55.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.894+0000 7f455e333700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 0x7f45580fd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60436/0 (socket says 172.21.15.190:60436) 2023-11-15T04:35:55.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.894+0000 7f455e333700 1 -- 172.21.15.190:0/3542091439 learned_addr learned my addr 172.21.15.190:0/3542091439 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:55.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.895+0000 7f455e333700 1 -- 172.21.15.190:0/3542091439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c005040 con 0x7f45580f59d0 2023-11-15T04:35:55.899 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.895+0000 7f455e333700 1 --2- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 0x7f45580fd390 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f454c013510 tx=0x7f454c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:55.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.895+0000 7f454affd700 1 -- 172.21.15.190:0/3542091439 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f454c009320 con 0x7f45580f59d0 2023-11-15T04:35:55.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.895+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45580fb9b0 con 0x7f45580f59d0 2023-11-15T04:35:55.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.895+0000 7f454affd700 1 -- 172.21.15.190:0/3542091439 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f454c013f00 con 0x7f45580f59d0 2023-11-15T04:35:55.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.895+0000 7f454affd700 1 -- 172.21.15.190:0/3542091439 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f454c014d90 con 0x7f45580f59d0 2023-11-15T04:35:55.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.895+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45580fbe40 con 0x7f45580f59d0 2023-11-15T04:35:55.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.896+0000 7f454affd700 1 -- 172.21.15.190:0/3542091439 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f454c0263a0 con 0x7f45580f59d0 2023-11-15T04:35:55.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.896+0000 7f454affd700 1 --2- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f454402ff60 0x7f4544032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:55.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.896+0000 7f454affd700 1 -- 172.21.15.190:0/3542091439 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f454c0095f0 con 0x7f45580f59d0 2023-11-15T04:35:55.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.896+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4558059100 con 0x7f45580f59d0 2023-11-15T04:35:55.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.896+0000 7f455db32700 1 --2- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f454402ff60 0x7f4544032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:55.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.896+0000 7f455db32700 1 --2- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f454402ff60 0x7f4544032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f45540096a0 tx=0x7f4554006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:55.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:55.901+0000 7f454affd700 1 -- 172.21.15.190:0/3542091439 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f454c012920 con 0x7f45580f59d0 2023-11-15T04:35:56.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.106+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4558045ad0 con 0x7f45580f59d0 2023-11-15T04:35:56.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.106+0000 7f454affd700 1 -- 172.21.15.190:0/3542091439 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f454c02b020 con 0x7f45580f59d0 2023-11-15T04:35:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:56.108 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:56.109 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f454402ff60 msgr2=0x7f4544032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:56.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 --2- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f454402ff60 0x7f4544032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f45540096a0 tx=0x7f4554006b40).stop 2023-11-15T04:35:56.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 msgr2=0x7f45580fd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:56.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 --2- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 0x7f45580fd390 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f454c013510 tx=0x7f454c00b040).stop 2023-11-15T04:35:56.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 shutdown_connections 2023-11-15T04:35:56.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 --2- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f454402ff60 0x7f4544032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:56.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 --2- 172.21.15.190:0/3542091439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45580f59d0 0x7f45580fd390 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:56.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 >> 172.21.15.190:0/3542091439 conn(0x7f45580f02e0 msgr2=0x7f45580f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:56.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 shutdown_connections 2023-11-15T04:35:56.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:56.108+0000 7f4560597700 1 -- 172.21.15.190:0/3542091439 wait complete. 2023-11-15T04:35:56.112 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:55.122198+0000 mgr.smithi047.zgysuz (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:56.107921+0000 mon.smithi047 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.190:0/3542091439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:35:57.493 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:35:57.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:35:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:57.122580+0000 mgr.smithi047.zgysuz (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:35:58.570 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.568+0000 7f2d68a95700 1 -- 172.21.15.190:0/3363033966 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 msgr2=0x7f2d640f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:58.571 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.568+0000 7f2d68a95700 1 --2- 172.21.15.190:0/3363033966 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 0x7f2d640f9190 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f2d58009fb0 tx=0x7f2d58009b30).stop 2023-11-15T04:35:58.571 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.568+0000 7f2d68a95700 1 -- 172.21.15.190:0/3363033966 shutdown_connections 2023-11-15T04:35:58.571 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.568+0000 7f2d68a95700 1 --2- 172.21.15.190:0/3363033966 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 0x7f2d640f9190 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:58.572 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.568+0000 7f2d68a95700 1 -- 172.21.15.190:0/3363033966 >> 172.21.15.190:0/3363033966 conn(0x7f2d640f0b70 msgr2=0x7f2d640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:58.572 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.568+0000 7f2d68a95700 1 -- 172.21.15.190:0/3363033966 shutdown_connections 2023-11-15T04:35:58.572 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.568+0000 7f2d68a95700 1 -- 172.21.15.190:0/3363033966 wait complete. 2023-11-15T04:35:58.572 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.569+0000 7f2d68a95700 1 Processor -- start 2023-11-15T04:35:58.572 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.569+0000 7f2d68a95700 1 -- start start 2023-11-15T04:35:58.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.569+0000 7f2d68a95700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 0x7f2d641066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:58.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.569+0000 7f2d68a95700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d64106bd0 con 0x7f2d640601e0 2023-11-15T04:35:58.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.569+0000 7f2d6259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 0x7f2d641066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:58.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.569+0000 7f2d6259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 0x7f2d641066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60462/0 (socket says 172.21.15.190:60462) 2023-11-15T04:35:58.573 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.569+0000 7f2d6259c700 1 -- 172.21.15.190:0/735696260 learned_addr learned my addr 172.21.15.190:0/735696260 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:35:58.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.570+0000 7f2d6259c700 1 -- 172.21.15.190:0/735696260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d58005040 con 0x7f2d640601e0 2023-11-15T04:35:58.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.570+0000 7f2d6259c700 1 --2- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 0x7f2d641066e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f2d58004dd0 tx=0x7f2d58009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:58.574 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.570+0000 7f2d52ffd700 1 -- 172.21.15.190:0/735696260 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d58018070 con 0x7f2d640601e0 2023-11-15T04:35:58.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.570+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d64106dd0 con 0x7f2d640601e0 2023-11-15T04:35:58.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.570+0000 7f2d52ffd700 1 -- 172.21.15.190:0/735696260 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d58013410 con 0x7f2d640601e0 2023-11-15T04:35:58.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.570+0000 7f2d52ffd700 1 -- 172.21.15.190:0/735696260 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d5801c3c0 con 0x7f2d640601e0 2023-11-15T04:35:58.575 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.570+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d64107260 con 0x7f2d640601e0 2023-11-15T04:35:58.577 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.571+0000 7f2d52ffd700 1 -- 172.21.15.190:0/735696260 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d5801d420 con 0x7f2d640601e0 2023-11-15T04:35:58.577 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.571+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d64045ad0 con 0x7f2d640601e0 2023-11-15T04:35:58.577 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.571+0000 7f2d52ffd700 1 --2- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2d4c02ff60 0x7f2d4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:35:58.577 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.571+0000 7f2d52ffd700 1 -- 172.21.15.190:0/735696260 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d58041310 con 0x7f2d640601e0 2023-11-15T04:35:58.578 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.571+0000 7f2d61d9b700 1 --2- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2d4c02ff60 0x7f2d4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:35:58.578 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.571+0000 7f2d61d9b700 1 --2- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2d4c02ff60 0x7f2d4c032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f2d540096a0 tx=0x7f2d54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:35:58.578 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.576+0000 7f2d52ffd700 1 -- 172.21.15.190:0/735696260 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d5801d820 con 0x7f2d640601e0 2023-11-15T04:35:58.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.781+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d641080b0 con 0x7f2d640601e0 2023-11-15T04:35:58.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.782+0000 7f2d52ffd700 1 -- 172.21.15.190:0/735696260 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d5801b370 con 0x7f2d640601e0 2023-11-15T04:35:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:35:58.783 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:35:58.784 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2d4c02ff60 msgr2=0x7f2d4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:58.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 --2- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2d4c02ff60 0x7f2d4c032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f2d540096a0 tx=0x7f2d54006b40).stop 2023-11-15T04:35:58.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 msgr2=0x7f2d641066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:35:58.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 --2- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 0x7f2d641066e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f2d58004dd0 tx=0x7f2d58009b30).stop 2023-11-15T04:35:58.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 shutdown_connections 2023-11-15T04:35:58.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 --2- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2d4c02ff60 0x7f2d4c032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:58.786 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 --2- 172.21.15.190:0/735696260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d640601e0 0x7f2d641066e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:35:58.786 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 >> 172.21.15.190:0/735696260 conn(0x7f2d640f0b70 msgr2=0x7f2d640f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:35:58.786 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 shutdown_connections 2023-11-15T04:35:58.786 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:35:58.783+0000 7f2d68a95700 1 -- 172.21.15.190:0/735696260 wait complete. 2023-11-15T04:35:58.787 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:35:59.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:58 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:35:58.783269+0000 mon.smithi047 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.190:0/735696260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:00.161 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:00.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:00.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:35:59 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:35:59.122925+0000 mgr.smithi047.zgysuz (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:01.335 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.333+0000 7f28753b3700 1 -- 172.21.15.190:0/2791669506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 msgr2=0x7f28700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:01.335 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.333+0000 7f28753b3700 1 --2- 172.21.15.190:0/2791669506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 0x7f28700f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f2864009fb0 tx=0x7f2864009b30).stop 2023-11-15T04:36:01.335 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.334+0000 7f28753b3700 1 -- 172.21.15.190:0/2791669506 shutdown_connections 2023-11-15T04:36:01.335 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.334+0000 7f28753b3700 1 --2- 172.21.15.190:0/2791669506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 0x7f28700f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:01.336 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.334+0000 7f28753b3700 1 -- 172.21.15.190:0/2791669506 >> 172.21.15.190:0/2791669506 conn(0x7f28700f0b70 msgr2=0x7f28700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:01.336 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.334+0000 7f28753b3700 1 -- 172.21.15.190:0/2791669506 shutdown_connections 2023-11-15T04:36:01.336 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.334+0000 7f28753b3700 1 -- 172.21.15.190:0/2791669506 wait complete. 2023-11-15T04:36:01.338 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.334+0000 7f28753b3700 1 Processor -- start 2023-11-15T04:36:01.338 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.334+0000 7f28753b3700 1 -- start start 2023-11-15T04:36:01.338 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.335+0000 7f28753b3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 0x7f2870105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:01.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.335+0000 7f28753b3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2870105af0 con 0x7f28700f5410 2023-11-15T04:36:01.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.335+0000 7f286effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 0x7f2870105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:01.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.335+0000 7f286effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 0x7f2870105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56398/0 (socket says 172.21.15.190:56398) 2023-11-15T04:36:01.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.335+0000 7f286effd700 1 -- 172.21.15.190:0/2167986353 learned_addr learned my addr 172.21.15.190:0/2167986353 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:01.339 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.335+0000 7f286effd700 1 -- 172.21.15.190:0/2167986353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2864005040 con 0x7f28700f5410 2023-11-15T04:36:01.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.336+0000 7f286effd700 1 --2- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 0x7f2870105600 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f2864013510 tx=0x7f286400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:01.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.336+0000 7f28577fe700 1 -- 172.21.15.190:0/2167986353 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2864009320 con 0x7f28700f5410 2023-11-15T04:36:01.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.336+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2870105cf0 con 0x7f28700f5410 2023-11-15T04:36:01.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.336+0000 7f28577fe700 1 -- 172.21.15.190:0/2167986353 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2864013f00 con 0x7f28700f5410 2023-11-15T04:36:01.340 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.336+0000 7f28577fe700 1 -- 172.21.15.190:0/2167986353 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2864014d90 con 0x7f28700f5410 2023-11-15T04:36:01.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.336+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2870106180 con 0x7f28700f5410 2023-11-15T04:36:01.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.337+0000 7f28577fe700 1 -- 172.21.15.190:0/2167986353 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28640095f0 con 0x7f28700f5410 2023-11-15T04:36:01.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.337+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2870045ad0 con 0x7f28700f5410 2023-11-15T04:36:01.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.337+0000 7f28577fe700 1 --2- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f285802ff60 0x7f2858032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:01.341 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.337+0000 7f28577fe700 1 -- 172.21.15.190:0/2167986353 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2864042d90 con 0x7f28700f5410 2023-11-15T04:36:01.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.337+0000 7f286e7fc700 1 --2- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f285802ff60 0x7f2858032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:01.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.337+0000 7f286e7fc700 1 --2- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f285802ff60 0x7f2858032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f28600096a0 tx=0x7f2860006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:01.342 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.342+0000 7f28577fe700 1 -- 172.21.15.190:0/2167986353 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f286401f7f0 con 0x7f28700f5410 2023-11-15T04:36:01.552 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.551+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28700fae60 con 0x7f28700f5410 2023-11-15T04:36:01.552 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.552+0000 7f28577fe700 1 -- 172.21.15.190:0/2167986353 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2864011290 con 0x7f28700f5410 2023-11-15T04:36:01.553 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:01.553 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:01.554 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f285802ff60 msgr2=0x7f2858032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:01.554 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 --2- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f285802ff60 0x7f2858032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f28600096a0 tx=0x7f2860006b40).stop 2023-11-15T04:36:01.554 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 msgr2=0x7f2870105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:01.555 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 --2- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 0x7f2870105600 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f2864013510 tx=0x7f286400b040).stop 2023-11-15T04:36:01.555 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 shutdown_connections 2023-11-15T04:36:01.555 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 --2- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f285802ff60 0x7f2858032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:01.555 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 --2- 172.21.15.190:0/2167986353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28700f5410 0x7f2870105600 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:01.556 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 >> 172.21.15.190:0/2167986353 conn(0x7f28700f0b70 msgr2=0x7f28700f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:01.556 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 shutdown_connections 2023-11-15T04:36:01.556 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:01.553+0000 7f28753b3700 1 -- 172.21.15.190:0/2167986353 wait complete. 2023-11-15T04:36:01.556 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:01.123253+0000 mgr.smithi047.zgysuz (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:01.553216+0000 mon.smithi047 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.190:0/2167986353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:02.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:02.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:04.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.096+0000 7f2bdb06a700 1 -- 172.21.15.190:0/2140412636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 msgr2=0x7f2bd40fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:04.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.096+0000 7f2bdb06a700 1 --2- 172.21.15.190:0/2140412636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 0x7f2bd40fb550 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f2bc8009fb0 tx=0x7f2bc8009b30).stop 2023-11-15T04:36:04.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.096+0000 7f2bdb06a700 1 -- 172.21.15.190:0/2140412636 shutdown_connections 2023-11-15T04:36:04.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.096+0000 7f2bdb06a700 1 --2- 172.21.15.190:0/2140412636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 0x7f2bd40fb550 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:04.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.096+0000 7f2bdb06a700 1 -- 172.21.15.190:0/2140412636 >> 172.21.15.190:0/2140412636 conn(0x7f2bd40f0b70 msgr2=0x7f2bd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:04.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.096+0000 7f2bdb06a700 1 -- 172.21.15.190:0/2140412636 shutdown_connections 2023-11-15T04:36:04.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.096+0000 7f2bdb06a700 1 -- 172.21.15.190:0/2140412636 wait complete. 2023-11-15T04:36:04.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.097+0000 7f2bdb06a700 1 Processor -- start 2023-11-15T04:36:04.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.097+0000 7f2bdb06a700 1 -- start start 2023-11-15T04:36:04.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.097+0000 7f2bdb06a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 0x7f2bd41088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:04.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.097+0000 7f2bdb06a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4108db0 con 0x7f2bd4060220 2023-11-15T04:36:04.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.097+0000 7f2bd8e06700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 0x7f2bd41088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:04.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.097+0000 7f2bd8e06700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 0x7f2bd41088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56414/0 (socket says 172.21.15.190:56414) 2023-11-15T04:36:04.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.097+0000 7f2bd8e06700 1 -- 172.21.15.190:0/4174471425 learned_addr learned my addr 172.21.15.190:0/4174471425 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:04.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.097+0000 7f2bd8e06700 1 -- 172.21.15.190:0/4174471425 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc8005040 con 0x7f2bd4060220 2023-11-15T04:36:04.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.098+0000 7f2bd8e06700 1 --2- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 0x7f2bd41088c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f2bc8013510 tx=0x7f2bc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:04.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.098+0000 7f2bd17fa700 1 -- 172.21.15.190:0/4174471425 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc8009320 con 0x7f2bd4060220 2023-11-15T04:36:04.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.098+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd4108fb0 con 0x7f2bd4060220 2023-11-15T04:36:04.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.098+0000 7f2bd17fa700 1 -- 172.21.15.190:0/4174471425 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bc8013f00 con 0x7f2bd4060220 2023-11-15T04:36:04.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.098+0000 7f2bd17fa700 1 -- 172.21.15.190:0/4174471425 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc8014dc0 con 0x7f2bd4060220 2023-11-15T04:36:04.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.098+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd4109440 con 0x7f2bd4060220 2023-11-15T04:36:04.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.099+0000 7f2bd17fa700 1 -- 172.21.15.190:0/4174471425 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bc80095f0 con 0x7f2bd4060220 2023-11-15T04:36:04.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.099+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd4045ad0 con 0x7f2bd4060220 2023-11-15T04:36:04.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.099+0000 7f2bd17fa700 1 --2- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2bc402ff60 0x7f2bc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:04.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.099+0000 7f2bd17fa700 1 -- 172.21.15.190:0/4174471425 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bc80410e0 con 0x7f2bd4060220 2023-11-15T04:36:04.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.099+0000 7f2bd3fff700 1 --2- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2bc402ff60 0x7f2bc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:04.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.099+0000 7f2bd3fff700 1 --2- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2bc402ff60 0x7f2bc4032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f2bc00096a0 tx=0x7f2bc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:04.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.104+0000 7f2bd17fa700 1 -- 172.21.15.190:0/4174471425 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bc8014420 con 0x7f2bd4060220 2023-11-15T04:36:04.314 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.313+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bd410a270 con 0x7f2bd4060220 2023-11-15T04:36:04.315 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.314+0000 7f2bd17fa700 1 -- 172.21.15.190:0/4174471425 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bc801c920 con 0x7f2bd4060220 2023-11-15T04:36:04.315 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:04.315 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:04.316 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.315+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2bc402ff60 msgr2=0x7f2bc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:04.316 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.315+0000 7f2bdb06a700 1 --2- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2bc402ff60 0x7f2bc4032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f2bc00096a0 tx=0x7f2bc0006b40).stop 2023-11-15T04:36:04.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.315+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 msgr2=0x7f2bd41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:04.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.315+0000 7f2bdb06a700 1 --2- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 0x7f2bd41088c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f2bc8013510 tx=0x7f2bc800b040).stop 2023-11-15T04:36:04.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.315+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 shutdown_connections 2023-11-15T04:36:04.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.315+0000 7f2bdb06a700 1 --2- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2bc402ff60 0x7f2bc4032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:04.317 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.315+0000 7f2bdb06a700 1 --2- 172.21.15.190:0/4174471425 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bd4060220 0x7f2bd41088c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:04.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.315+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 >> 172.21.15.190:0/4174471425 conn(0x7f2bd40f0b70 msgr2=0x7f2bd40f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:04.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.315+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 shutdown_connections 2023-11-15T04:36:04.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:04.316+0000 7f2bdb06a700 1 -- 172.21.15.190:0/4174471425 wait complete. 2023-11-15T04:36:04.319 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:03.123547+0000 mgr.smithi047.zgysuz (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:05.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:05 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:04.315168+0000 mon.smithi047 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.190:0/4174471425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:05.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:05.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:05.123887+0000 mgr.smithi047.zgysuz (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:05.220925+0000 mon.smithi047 (mon.0) 281 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:36:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:05.222191+0000 mon.smithi047 (mon.0) 282 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:36:06.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.878+0000 7f7ee6a74700 1 -- 172.21.15.190:0/3353997243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 msgr2=0x7f7ee00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:06.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.878+0000 7f7ee6a74700 1 --2- 172.21.15.190:0/3353997243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 0x7f7ee00f6930 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f7ed4009fb0 tx=0x7f7ed4009b30).stop 2023-11-15T04:36:06.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.878+0000 7f7ee6a74700 1 -- 172.21.15.190:0/3353997243 shutdown_connections 2023-11-15T04:36:06.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.878+0000 7f7ee6a74700 1 --2- 172.21.15.190:0/3353997243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 0x7f7ee00f6930 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:06.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.878+0000 7f7ee6a74700 1 -- 172.21.15.190:0/3353997243 >> 172.21.15.190:0/3353997243 conn(0x7f7ee00f1c70 msgr2=0x7f7ee00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:06.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.878+0000 7f7ee6a74700 1 -- 172.21.15.190:0/3353997243 shutdown_connections 2023-11-15T04:36:06.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.878+0000 7f7ee6a74700 1 -- 172.21.15.190:0/3353997243 wait complete. 2023-11-15T04:36:06.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.879+0000 7f7ee6a74700 1 Processor -- start 2023-11-15T04:36:06.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.879+0000 7f7ee6a74700 1 -- start start 2023-11-15T04:36:06.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.879+0000 7f7ee6a74700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 0x7f7ee0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:06.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.879+0000 7f7ee6a74700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee0106c20 con 0x7f7ee00f6510 2023-11-15T04:36:06.882 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.879+0000 7f7ee4810700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 0x7f7ee0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:06.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.879+0000 7f7ee4810700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 0x7f7ee0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56432/0 (socket says 172.21.15.190:56432) 2023-11-15T04:36:06.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.879+0000 7f7ee4810700 1 -- 172.21.15.190:0/2992358646 learned_addr learned my addr 172.21.15.190:0/2992358646 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:06.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.880+0000 7f7ee4810700 1 -- 172.21.15.190:0/2992358646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed4005040 con 0x7f7ee00f6510 2023-11-15T04:36:06.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.880+0000 7f7ee4810700 1 --2- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 0x7f7ee0106730 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f7ed4013510 tx=0x7f7ed400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:06.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.880+0000 7f7edd7fa700 1 -- 172.21.15.190:0/2992358646 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ed4009320 con 0x7f7ee00f6510 2023-11-15T04:36:06.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.880+0000 7f7edd7fa700 1 -- 172.21.15.190:0/2992358646 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ed4013f00 con 0x7f7ee00f6510 2023-11-15T04:36:06.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.880+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee0106e20 con 0x7f7ee00f6510 2023-11-15T04:36:06.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.880+0000 7f7edd7fa700 1 -- 172.21.15.190:0/2992358646 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ed4014d90 con 0x7f7ee00f6510 2023-11-15T04:36:06.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.880+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee01072b0 con 0x7f7ee00f6510 2023-11-15T04:36:06.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.881+0000 7f7edd7fa700 1 -- 172.21.15.190:0/2992358646 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ed4014420 con 0x7f7ee00f6510 2023-11-15T04:36:06.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.881+0000 7f7edd7fa700 1 --2- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ed002ff60 0x7f7ed0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:06.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.881+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee0045ad0 con 0x7f7ee00f6510 2023-11-15T04:36:06.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.881+0000 7f7edd7fa700 1 -- 172.21.15.190:0/2992358646 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ed4041000 con 0x7f7ee00f6510 2023-11-15T04:36:06.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.881+0000 7f7edffff700 1 --2- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ed002ff60 0x7f7ed0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:06.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.881+0000 7f7edffff700 1 --2- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ed002ff60 0x7f7ed0032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7ecc0096a0 tx=0x7f7ecc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:06.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:06.886+0000 7f7edd7fa700 1 -- 172.21.15.190:0/2992358646 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ed40095f0 con 0x7f7ee00f6510 2023-11-15T04:36:07.094 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.093+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ee00fc050 con 0x7f7ee00f6510 2023-11-15T04:36:07.095 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.094+0000 7f7edd7fa700 1 -- 172.21.15.190:0/2992358646 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ed403f090 con 0x7f7ee00f6510 2023-11-15T04:36:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:07.096 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:07.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ed002ff60 msgr2=0x7f7ed0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:07.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 --2- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ed002ff60 0x7f7ed0032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7ecc0096a0 tx=0x7f7ecc006b40).stop 2023-11-15T04:36:07.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 msgr2=0x7f7ee0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:07.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 --2- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 0x7f7ee0106730 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f7ed4013510 tx=0x7f7ed400b040).stop 2023-11-15T04:36:07.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 shutdown_connections 2023-11-15T04:36:07.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 --2- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ed002ff60 0x7f7ed0032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:07.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 --2- 172.21.15.190:0/2992358646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ee00f6510 0x7f7ee0106730 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:07.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 >> 172.21.15.190:0/2992358646 conn(0x7f7ee00f1c70 msgr2=0x7f7ee00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:07.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 shutdown_connections 2023-11-15T04:36:07.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:07.096+0000 7f7ee6a74700 1 -- 172.21.15.190:0/2992358646 wait complete. 2023-11-15T04:36:07.099 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:08.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:07.095767+0000 mon.smithi047 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.190:0/2992358646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:08.008 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:07.124165+0000 mgr.smithi047.zgysuz (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:08.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:08.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:09.585 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.583+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2786210354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 msgr2=0x7ff2380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:09.586 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.583+0000 7ff23ff2b700 1 --2- 172.21.15.190:0/2786210354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 0x7ff2380f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7ff234009fb0 tx=0x7ff234009b30).stop 2023-11-15T04:36:09.586 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.583+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2786210354 shutdown_connections 2023-11-15T04:36:09.586 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.583+0000 7ff23ff2b700 1 --2- 172.21.15.190:0/2786210354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 0x7ff2380f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:09.586 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.583+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2786210354 >> 172.21.15.190:0/2786210354 conn(0x7ff2380f0b70 msgr2=0x7ff2380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:09.587 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.583+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2786210354 shutdown_connections 2023-11-15T04:36:09.587 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.583+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2786210354 wait complete. 2023-11-15T04:36:09.587 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.583+0000 7ff23ff2b700 1 Processor -- start 2023-11-15T04:36:09.587 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.583+0000 7ff23ff2b700 1 -- start start 2023-11-15T04:36:09.587 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff23ff2b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 0x7ff238109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:09.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff23ff2b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff238060220 con 0x7ff2380f5410 2023-11-15T04:36:09.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff23dcc7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 0x7ff238109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:09.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff23dcc7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 0x7ff238109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56448/0 (socket says 172.21.15.190:56448) 2023-11-15T04:36:09.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff23dcc7700 1 -- 172.21.15.190:0/2046892560 learned_addr learned my addr 172.21.15.190:0/2046892560 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:09.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff23dcc7700 1 -- 172.21.15.190:0/2046892560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff234005040 con 0x7ff2380f5410 2023-11-15T04:36:09.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff23dcc7700 1 --2- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 0x7ff238109450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7ff234013510 tx=0x7ff23400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:09.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff22e7fc700 1 -- 172.21.15.190:0/2046892560 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff234009320 con 0x7ff2380f5410 2023-11-15T04:36:09.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff238060420 con 0x7ff2380f5410 2023-11-15T04:36:09.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff22e7fc700 1 -- 172.21.15.190:0/2046892560 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff234013f00 con 0x7ff2380f5410 2023-11-15T04:36:09.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.584+0000 7ff22e7fc700 1 -- 172.21.15.190:0/2046892560 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff234014d90 con 0x7ff2380f5410 2023-11-15T04:36:09.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.585+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff238109b00 con 0x7ff2380f5410 2023-11-15T04:36:09.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.585+0000 7ff22e7fc700 1 -- 172.21.15.190:0/2046892560 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff234014420 con 0x7ff2380f5410 2023-11-15T04:36:09.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.585+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff238059100 con 0x7ff2380f5410 2023-11-15T04:36:09.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.585+0000 7ff22e7fc700 1 --2- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff22402ff60 0x7ff224032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:09.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.585+0000 7ff22e7fc700 1 -- 172.21.15.190:0/2046892560 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff234041050 con 0x7ff2380f5410 2023-11-15T04:36:09.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.586+0000 7ff23d4c6700 1 --2- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff22402ff60 0x7ff224032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:09.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.586+0000 7ff23d4c6700 1 --2- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff22402ff60 0x7ff224032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7ff2280096a0 tx=0x7ff228006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:09.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.590+0000 7ff22e7fc700 1 -- 172.21.15.190:0/2046892560 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2340095f0 con 0x7ff2380f5410 2023-11-15T04:36:09.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.799+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff238045ad0 con 0x7ff2380f5410 2023-11-15T04:36:09.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.801+0000 7ff22e7fc700 1 -- 172.21.15.190:0/2046892560 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff23401db70 con 0x7ff2380f5410 2023-11-15T04:36:09.801 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:09.802 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:09.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff22402ff60 msgr2=0x7ff224032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:09.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 --2- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff22402ff60 0x7ff224032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7ff2280096a0 tx=0x7ff228006b40).stop 2023-11-15T04:36:09.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 msgr2=0x7ff238109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:09.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 --2- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 0x7ff238109450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7ff234013510 tx=0x7ff23400b040).stop 2023-11-15T04:36:09.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 shutdown_connections 2023-11-15T04:36:09.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 --2- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff22402ff60 0x7ff224032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:09.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 --2- 172.21.15.190:0/2046892560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2380f5410 0x7ff238109450 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:09.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 >> 172.21.15.190:0/2046892560 conn(0x7ff2380f0b70 msgr2=0x7ff2380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:09.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 shutdown_connections 2023-11-15T04:36:09.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:09.802+0000 7ff23ff2b700 1 -- 172.21.15.190:0/2046892560 wait complete. 2023-11-15T04:36:09.805 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:10.510 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:09.124517+0000 mgr.smithi047.zgysuz (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:10.511 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:09.801526+0000 mon.smithi047 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.190:0/2046892560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:11.186 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:11.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:12.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.373+0000 7f529e463700 1 -- 172.21.15.190:0/2680104838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 msgr2=0x7f52980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:12.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.373+0000 7f529e463700 1 --2- 172.21.15.190:0/2680104838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 0x7f52980f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f5288009fb0 tx=0x7f5288009b30).stop 2023-11-15T04:36:12.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.374+0000 7f529e463700 1 -- 172.21.15.190:0/2680104838 shutdown_connections 2023-11-15T04:36:12.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.374+0000 7f529e463700 1 --2- 172.21.15.190:0/2680104838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 0x7f52980f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:12.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.374+0000 7f529e463700 1 -- 172.21.15.190:0/2680104838 >> 172.21.15.190:0/2680104838 conn(0x7f52980f0b70 msgr2=0x7f52980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:12.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.374+0000 7f529e463700 1 -- 172.21.15.190:0/2680104838 shutdown_connections 2023-11-15T04:36:12.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.374+0000 7f529e463700 1 -- 172.21.15.190:0/2680104838 wait complete. 2023-11-15T04:36:12.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.374+0000 7f529e463700 1 Processor -- start 2023-11-15T04:36:12.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.374+0000 7f529e463700 1 -- start start 2023-11-15T04:36:12.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.374+0000 7f529e463700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 0x7f5298109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:12.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.374+0000 7f529e463700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5298060220 con 0x7f52980f5410 2023-11-15T04:36:12.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f5297fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 0x7f5298109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:12.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f5297fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 0x7f5298109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:43866/0 (socket says 172.21.15.190:43866) 2023-11-15T04:36:12.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f5297fff700 1 -- 172.21.15.190:0/2957436458 learned_addr learned my addr 172.21.15.190:0/2957436458 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:12.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f5297fff700 1 -- 172.21.15.190:0/2957436458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5288005040 con 0x7f52980f5410 2023-11-15T04:36:12.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f5297fff700 1 --2- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 0x7f5298109450 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f5288013510 tx=0x7f528800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:12.379 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f5294ff9700 1 -- 172.21.15.190:0/2957436458 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5288009320 con 0x7f52980f5410 2023-11-15T04:36:12.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5298060420 con 0x7f52980f5410 2023-11-15T04:36:12.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f5294ff9700 1 -- 172.21.15.190:0/2957436458 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5288013f00 con 0x7f52980f5410 2023-11-15T04:36:12.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f5294ff9700 1 -- 172.21.15.190:0/2957436458 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5288014d90 con 0x7f52980f5410 2023-11-15T04:36:12.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.375+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5298109b00 con 0x7f52980f5410 2023-11-15T04:36:12.381 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.376+0000 7f5294ff9700 1 -- 172.21.15.190:0/2957436458 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52880263a0 con 0x7f52980f5410 2023-11-15T04:36:12.381 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.376+0000 7f5294ff9700 1 --2- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f528402ff60 0x7f5284032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:12.382 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.376+0000 7f5294ff9700 1 -- 172.21.15.190:0/2957436458 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5288040fe0 con 0x7f52980f5410 2023-11-15T04:36:12.382 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.376+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5298059100 con 0x7f52980f5410 2023-11-15T04:36:12.382 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.376+0000 7f52977fe700 1 --2- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f528402ff60 0x7f5284032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:12.382 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.377+0000 7f52977fe700 1 --2- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f528402ff60 0x7f5284032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f52800096a0 tx=0x7f5280006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:12.383 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.382+0000 7f5294ff9700 1 -- 172.21.15.190:0/2957436458 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5288012920 con 0x7f52980f5410 2023-11-15T04:36:12.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:12 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:11.124820+0000 mgr.smithi047.zgysuz (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:12.587 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.586+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5298045ad0 con 0x7f52980f5410 2023-11-15T04:36:12.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.587+0000 7f5294ff9700 1 -- 172.21.15.190:0/2957436458 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f528802b020 con 0x7f52980f5410 2023-11-15T04:36:12.588 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:12.588 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:12.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f528402ff60 msgr2=0x7f5284032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:12.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 --2- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f528402ff60 0x7f5284032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f52800096a0 tx=0x7f5280006b40).stop 2023-11-15T04:36:12.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 msgr2=0x7f5298109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:12.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 --2- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 0x7f5298109450 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f5288013510 tx=0x7f528800b040).stop 2023-11-15T04:36:12.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 shutdown_connections 2023-11-15T04:36:12.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 --2- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f528402ff60 0x7f5284032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:12.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 --2- 172.21.15.190:0/2957436458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52980f5410 0x7f5298109450 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:12.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 >> 172.21.15.190:0/2957436458 conn(0x7f52980f0b70 msgr2=0x7f52980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:12.591 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 shutdown_connections 2023-11-15T04:36:12.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:12.589+0000 7f529e463700 1 -- 172.21.15.190:0/2957436458 wait complete. 2023-11-15T04:36:12.592 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:13.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:13 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:12.588442+0000 mon.smithi047 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.190:0/2957436458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:13.968 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:13.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:13.125201+0000 mgr.smithi047.zgysuz (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:15.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 -- 172.21.15.190:0/4223020559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 msgr2=0x7f99ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:15.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 --2- 172.21.15.190:0/4223020559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 0x7f99ac0f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f999c009fb0 tx=0x7f999c009b30).stop 2023-11-15T04:36:15.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 -- 172.21.15.190:0/4223020559 shutdown_connections 2023-11-15T04:36:15.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 --2- 172.21.15.190:0/4223020559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 0x7f99ac0f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:15.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 -- 172.21.15.190:0/4223020559 >> 172.21.15.190:0/4223020559 conn(0x7f99ac0f1c70 msgr2=0x7f99ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:15.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 -- 172.21.15.190:0/4223020559 shutdown_connections 2023-11-15T04:36:15.119 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 -- 172.21.15.190:0/4223020559 wait complete. 2023-11-15T04:36:15.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 Processor -- start 2023-11-15T04:36:15.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 -- start start 2023-11-15T04:36:15.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 0x7f99ac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:15.120 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.117+0000 7f99b222e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac106c20 con 0x7f99ac0f6510 2023-11-15T04:36:15.122 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f99ab7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 0x7f99ac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:15.122 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f99ab7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 0x7f99ac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:43882/0 (socket says 172.21.15.190:43882) 2023-11-15T04:36:15.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f99ab7fe700 1 -- 172.21.15.190:0/63907338 learned_addr learned my addr 172.21.15.190:0/63907338 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:15.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f99ab7fe700 1 -- 172.21.15.190:0/63907338 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c005040 con 0x7f99ac0f6510 2023-11-15T04:36:15.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f99ab7fe700 1 --2- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 0x7f99ac106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f999c014000 tx=0x7f999c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:15.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f9993fff700 1 -- 172.21.15.190:0/63907338 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f999c0095f0 con 0x7f99ac0f6510 2023-11-15T04:36:15.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ac106e20 con 0x7f99ac0f6510 2023-11-15T04:36:15.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f9993fff700 1 -- 172.21.15.190:0/63907338 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f999c007e70 con 0x7f99ac0f6510 2023-11-15T04:36:15.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f9993fff700 1 -- 172.21.15.190:0/63907338 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f999c014ea0 con 0x7f99ac0f6510 2023-11-15T04:36:15.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.118+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ac1072b0 con 0x7f99ac0f6510 2023-11-15T04:36:15.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.120+0000 7f9993fff700 1 -- 172.21.15.190:0/63907338 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f999c025070 con 0x7f99ac0f6510 2023-11-15T04:36:15.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.120+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ac045ad0 con 0x7f99ac0f6510 2023-11-15T04:36:15.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.120+0000 7f9993fff700 1 --2- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f999402ff60 0x7f9994032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:15.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.120+0000 7f9993fff700 1 -- 172.21.15.190:0/63907338 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f999c040940 con 0x7f99ac0f6510 2023-11-15T04:36:15.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.120+0000 7f99aaffd700 1 --2- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f999402ff60 0x7f9994032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:15.125 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.120+0000 7f99aaffd700 1 --2- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f999402ff60 0x7f9994032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f99a00096a0 tx=0x7f99a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:15.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.125+0000 7f9993fff700 1 -- 172.21.15.190:0/63907338 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f999c0079c0 con 0x7f99ac0f6510 2023-11-15T04:36:15.330 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.330+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99ac0fbf60 con 0x7f99ac0f6510 2023-11-15T04:36:15.331 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.331+0000 7f9993fff700 1 -- 172.21.15.190:0/63907338 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f999c00f300 con 0x7f99ac0f6510 2023-11-15T04:36:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:15.331 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:15.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f999402ff60 msgr2=0x7f9994032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:15.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 --2- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f999402ff60 0x7f9994032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f99a00096a0 tx=0x7f99a0006b40).stop 2023-11-15T04:36:15.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 msgr2=0x7f99ac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:15.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 --2- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 0x7f99ac106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f999c014000 tx=0x7f999c00b040).stop 2023-11-15T04:36:15.333 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 shutdown_connections 2023-11-15T04:36:15.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 --2- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f999402ff60 0x7f9994032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:15.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 --2- 172.21.15.190:0/63907338 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99ac0f6510 0x7f99ac106730 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:15.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 >> 172.21.15.190:0/63907338 conn(0x7f99ac0f1c70 msgr2=0x7f99ac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:15.334 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 shutdown_connections 2023-11-15T04:36:15.335 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:15.333+0000 7f99b222e700 1 -- 172.21.15.190:0/63907338 wait complete. 2023-11-15T04:36:15.335 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:15.125470+0000 mgr.smithi047.zgysuz (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:15.331484+0000 mon.smithi047 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.190:0/63907338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:16.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:16.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:17.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.885+0000 7fb4433da700 1 -- 172.21.15.190:0/1400553037 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 msgr2=0x7fb43c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:17.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.885+0000 7fb4433da700 1 --2- 172.21.15.190:0/1400553037 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 0x7fb43c0f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fb430009fb0 tx=0x7fb430009b30).stop 2023-11-15T04:36:17.890 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.885+0000 7fb4433da700 1 -- 172.21.15.190:0/1400553037 shutdown_connections 2023-11-15T04:36:17.890 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.885+0000 7fb4433da700 1 --2- 172.21.15.190:0/1400553037 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 0x7fb43c0f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:17.890 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.885+0000 7fb4433da700 1 -- 172.21.15.190:0/1400553037 >> 172.21.15.190:0/1400553037 conn(0x7fb43c0f0b70 msgr2=0x7fb43c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:17.890 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.885+0000 7fb4433da700 1 -- 172.21.15.190:0/1400553037 shutdown_connections 2023-11-15T04:36:17.891 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.885+0000 7fb4433da700 1 -- 172.21.15.190:0/1400553037 wait complete. 2023-11-15T04:36:17.891 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.886+0000 7fb4433da700 1 Processor -- start 2023-11-15T04:36:17.891 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.886+0000 7fb4433da700 1 -- start start 2023-11-15T04:36:17.891 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.886+0000 7fb4433da700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 0x7fb43c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:17.891 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.886+0000 7fb4433da700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c060240 con 0x7fb43c0f5410 2023-11-15T04:36:17.892 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.886+0000 7fb441176700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 0x7fb43c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:17.892 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.886+0000 7fb441176700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 0x7fb43c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:43914/0 (socket says 172.21.15.190:43914) 2023-11-15T04:36:17.892 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.886+0000 7fb441176700 1 -- 172.21.15.190:0/4108076736 learned_addr learned my addr 172.21.15.190:0/4108076736 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:17.892 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.887+0000 7fb441176700 1 -- 172.21.15.190:0/4108076736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb430005040 con 0x7fb43c0f5410 2023-11-15T04:36:17.892 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.887+0000 7fb441176700 1 --2- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 0x7fb43c109430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fb430014000 tx=0x7fb43000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:17.894 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.887+0000 7fb42dffb700 1 -- 172.21.15.190:0/4108076736 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4300095f0 con 0x7fb43c0f5410 2023-11-15T04:36:17.894 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.887+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43c060440 con 0x7fb43c0f5410 2023-11-15T04:36:17.894 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.887+0000 7fb42dffb700 1 -- 172.21.15.190:0/4108076736 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb430007e70 con 0x7fb43c0f5410 2023-11-15T04:36:17.895 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.887+0000 7fb42dffb700 1 -- 172.21.15.190:0/4108076736 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb430014ea0 con 0x7fb43c0f5410 2023-11-15T04:36:17.895 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.887+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43c109ae0 con 0x7fb43c0f5410 2023-11-15T04:36:17.895 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.888+0000 7fb42dffb700 1 -- 172.21.15.190:0/4108076736 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb430025070 con 0x7fb43c0f5410 2023-11-15T04:36:17.895 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.888+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43c045ad0 con 0x7fb43c0f5410 2023-11-15T04:36:17.895 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.888+0000 7fb42dffb700 1 --2- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb42802ff60 0x7fb428032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:17.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.888+0000 7fb42dffb700 1 -- 172.21.15.190:0/4108076736 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4300409b0 con 0x7fb43c0f5410 2023-11-15T04:36:17.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.888+0000 7fb440975700 1 --2- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb42802ff60 0x7fb428032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:17.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.889+0000 7fb440975700 1 --2- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb42802ff60 0x7fb428032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb4380096a0 tx=0x7fb438006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:17.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:17.893+0000 7fb42dffb700 1 -- 172.21.15.190:0/4108076736 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb43000f4e0 con 0x7fb43c0f5410 2023-11-15T04:36:18.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:17.125810+0000 mgr.smithi047.zgysuz (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:18.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.099+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb43c0fae60 con 0x7fb43c0f5410 2023-11-15T04:36:18.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.099+0000 7fb42dffb700 1 -- 172.21.15.190:0/4108076736 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb43000f300 con 0x7fb43c0f5410 2023-11-15T04:36:18.100 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:18.100 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:18.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb42802ff60 msgr2=0x7fb428032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:18.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 --2- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb42802ff60 0x7fb428032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb4380096a0 tx=0x7fb438006b40).stop 2023-11-15T04:36:18.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 msgr2=0x7fb43c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:18.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 --2- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 0x7fb43c109430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fb430014000 tx=0x7fb43000b040).stop 2023-11-15T04:36:18.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 shutdown_connections 2023-11-15T04:36:18.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 --2- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb42802ff60 0x7fb428032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:18.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 --2- 172.21.15.190:0/4108076736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0f5410 0x7fb43c109430 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:18.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 >> 172.21.15.190:0/4108076736 conn(0x7fb43c0f0b70 msgr2=0x7fb43c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:18.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 shutdown_connections 2023-11-15T04:36:18.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:18.101+0000 7fb4433da700 1 -- 172.21.15.190:0/4108076736 wait complete. 2023-11-15T04:36:18.104 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:19.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:18.099790+0000 mon.smithi047 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.190:0/4108076736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:19.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:19.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:20.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:19 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:19.126198+0000 mgr.smithi047.zgysuz (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:20.665 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.665+0000 7f71bd80d700 1 -- 172.21.15.190:0/2349554029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 msgr2=0x7f71b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:20.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.665+0000 7f71bd80d700 1 --2- 172.21.15.190:0/2349554029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 0x7f71b80f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f71ac009fb0 tx=0x7f71ac009b30).stop 2023-11-15T04:36:20.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.665+0000 7f71bd80d700 1 -- 172.21.15.190:0/2349554029 shutdown_connections 2023-11-15T04:36:20.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.665+0000 7f71bd80d700 1 --2- 172.21.15.190:0/2349554029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 0x7f71b80f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:20.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.665+0000 7f71bd80d700 1 -- 172.21.15.190:0/2349554029 >> 172.21.15.190:0/2349554029 conn(0x7f71b80f0b70 msgr2=0x7f71b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:20.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.665+0000 7f71bd80d700 1 -- 172.21.15.190:0/2349554029 shutdown_connections 2023-11-15T04:36:20.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.665+0000 7f71bd80d700 1 -- 172.21.15.190:0/2349554029 wait complete. 2023-11-15T04:36:20.669 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.666+0000 7f71bd80d700 1 Processor -- start 2023-11-15T04:36:20.669 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.666+0000 7f71bd80d700 1 -- start start 2023-11-15T04:36:20.669 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.666+0000 7f71bd80d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 0x7f71b8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:20.669 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.666+0000 7f71bd80d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b8060220 con 0x7f71b80f5410 2023-11-15T04:36:20.669 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.666+0000 7f71b6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 0x7f71b8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:20.670 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.666+0000 7f71b6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 0x7f71b8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:42404/0 (socket says 172.21.15.190:42404) 2023-11-15T04:36:20.670 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.666+0000 7f71b6ffd700 1 -- 172.21.15.190:0/1783200579 learned_addr learned my addr 172.21.15.190:0/1783200579 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:20.670 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.667+0000 7f71b6ffd700 1 -- 172.21.15.190:0/1783200579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac005040 con 0x7f71b80f5410 2023-11-15T04:36:20.670 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.667+0000 7f71b6ffd700 1 --2- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 0x7f71b8109450 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f71ac013510 tx=0x7f71ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:20.670 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.667+0000 7f719f7fe700 1 -- 172.21.15.190:0/1783200579 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71ac009320 con 0x7f71b80f5410 2023-11-15T04:36:20.671 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.667+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b8060420 con 0x7f71b80f5410 2023-11-15T04:36:20.671 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.667+0000 7f719f7fe700 1 -- 172.21.15.190:0/1783200579 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71ac013f00 con 0x7f71b80f5410 2023-11-15T04:36:20.671 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.667+0000 7f719f7fe700 1 -- 172.21.15.190:0/1783200579 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71ac014d90 con 0x7f71b80f5410 2023-11-15T04:36:20.671 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.667+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b8109b00 con 0x7f71b80f5410 2023-11-15T04:36:20.671 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.668+0000 7f719f7fe700 1 -- 172.21.15.190:0/1783200579 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71ac014420 con 0x7f71b80f5410 2023-11-15T04:36:20.672 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.668+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b8059100 con 0x7f71b80f5410 2023-11-15T04:36:20.672 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.668+0000 7f719f7fe700 1 --2- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f71a002ffb0 0x7f71a0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:20.672 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.668+0000 7f719f7fe700 1 -- 172.21.15.190:0/1783200579 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71ac0410c0 con 0x7f71b80f5410 2023-11-15T04:36:20.672 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.669+0000 7f71b67fc700 1 --2- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f71a002ffb0 0x7f71a0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:20.672 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.669+0000 7f71b67fc700 1 --2- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f71a002ffb0 0x7f71a0032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f71a80096a0 tx=0x7f71a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:20.673 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.673+0000 7f719f7fe700 1 -- 172.21.15.190:0/1783200579 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71ac0095f0 con 0x7f71b80f5410 2023-11-15T04:36:20.877 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.877+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71b8045ad0 con 0x7f71b80f5410 2023-11-15T04:36:20.878 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.878+0000 7f719f7fe700 1 -- 172.21.15.190:0/1783200579 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f71ac01db70 con 0x7f71b80f5410 2023-11-15T04:36:20.878 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:20.879 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:20.879 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f71a002ffb0 msgr2=0x7f71a0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:20.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 --2- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f71a002ffb0 0x7f71a0032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f71a80096a0 tx=0x7f71a8006b40).stop 2023-11-15T04:36:20.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 msgr2=0x7f71b8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:20.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 --2- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 0x7f71b8109450 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f71ac013510 tx=0x7f71ac00b040).stop 2023-11-15T04:36:20.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 shutdown_connections 2023-11-15T04:36:20.880 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 --2- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f71a002ffb0 0x7f71a0032470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:20.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 --2- 172.21.15.190:0/1783200579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71b80f5410 0x7f71b8109450 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:20.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 >> 172.21.15.190:0/1783200579 conn(0x7f71b80f0b70 msgr2=0x7f71b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:20.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 shutdown_connections 2023-11-15T04:36:20.881 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:20.879+0000 7f71bd80d700 1 -- 172.21.15.190:0/1783200579 wait complete. 2023-11-15T04:36:20.882 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:21.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:20 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:20.878453+0000 mon.smithi047 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.190:0/1783200579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:22.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:21 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:21.126613+0000 mgr.smithi047.zgysuz (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:22.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:22.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:23.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.386+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/820186191 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 msgr2=0x7fe9cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:23.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.386+0000 7fe9d2ee4700 1 --2- 172.21.15.190:0/820186191 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 0x7fe9cc0f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fe9c0009fb0 tx=0x7fe9c0009b30).stop 2023-11-15T04:36:23.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.386+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/820186191 shutdown_connections 2023-11-15T04:36:23.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.386+0000 7fe9d2ee4700 1 --2- 172.21.15.190:0/820186191 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 0x7fe9cc0f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:23.387 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.386+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/820186191 >> 172.21.15.190:0/820186191 conn(0x7fe9cc0f0b70 msgr2=0x7fe9cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:23.387 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.386+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/820186191 shutdown_connections 2023-11-15T04:36:23.387 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.386+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/820186191 wait complete. 2023-11-15T04:36:23.389 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.386+0000 7fe9d2ee4700 1 Processor -- start 2023-11-15T04:36:23.389 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.386+0000 7fe9d2ee4700 1 -- start start 2023-11-15T04:36:23.389 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.387+0000 7fe9d2ee4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 0x7fe9cc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:23.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.387+0000 7fe9d2ee4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc105b20 con 0x7fe9cc0f5410 2023-11-15T04:36:23.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.387+0000 7fe9d0c80700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 0x7fe9cc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:23.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.387+0000 7fe9d0c80700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 0x7fe9cc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:42416/0 (socket says 172.21.15.190:42416) 2023-11-15T04:36:23.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.387+0000 7fe9d0c80700 1 -- 172.21.15.190:0/1286904100 learned_addr learned my addr 172.21.15.190:0/1286904100 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:23.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.388+0000 7fe9d0c80700 1 -- 172.21.15.190:0/1286904100 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0005040 con 0x7fe9cc0f5410 2023-11-15T04:36:23.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.388+0000 7fe9d0c80700 1 --2- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 0x7fe9cc105630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fe9c0013510 tx=0x7fe9c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:23.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.388+0000 7fe9c97fa700 1 -- 172.21.15.190:0/1286904100 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9c0009320 con 0x7fe9cc0f5410 2023-11-15T04:36:23.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.388+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc105d20 con 0x7fe9cc0f5410 2023-11-15T04:36:23.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.388+0000 7fe9c97fa700 1 -- 172.21.15.190:0/1286904100 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9c0013f00 con 0x7fe9cc0f5410 2023-11-15T04:36:23.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.388+0000 7fe9c97fa700 1 -- 172.21.15.190:0/1286904100 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9c0014d90 con 0x7fe9cc0f5410 2023-11-15T04:36:23.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.388+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc1061b0 con 0x7fe9cc0f5410 2023-11-15T04:36:23.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.389+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc045ad0 con 0x7fe9cc0f5410 2023-11-15T04:36:23.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.389+0000 7fe9c97fa700 1 -- 172.21.15.190:0/1286904100 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9c00095f0 con 0x7fe9cc0f5410 2023-11-15T04:36:23.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.389+0000 7fe9c97fa700 1 --2- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe9bc02ff60 0x7fe9bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:23.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.389+0000 7fe9c97fa700 1 -- 172.21.15.190:0/1286904100 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9c0041090 con 0x7fe9cc0f5410 2023-11-15T04:36:23.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.389+0000 7fe9cbfff700 1 --2- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe9bc02ff60 0x7fe9bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:23.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.390+0000 7fe9cbfff700 1 --2- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe9bc02ff60 0x7fe9bc032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe9b80096a0 tx=0x7fe9b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:23.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.394+0000 7fe9c97fa700 1 -- 172.21.15.190:0/1286904100 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9c0014420 con 0x7fe9cc0f5410 2023-11-15T04:36:23.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.602+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9cc0fae60 con 0x7fe9cc0f5410 2023-11-15T04:36:23.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.602+0000 7fe9c97fa700 1 -- 172.21.15.190:0/1286904100 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9c001d7f0 con 0x7fe9cc0f5410 2023-11-15T04:36:23.603 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:23.603 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:23.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe9bc02ff60 msgr2=0x7fe9bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:23.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 --2- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe9bc02ff60 0x7fe9bc032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe9b80096a0 tx=0x7fe9b8006b40).stop 2023-11-15T04:36:23.604 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 msgr2=0x7fe9cc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:23.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 --2- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 0x7fe9cc105630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fe9c0013510 tx=0x7fe9c000b040).stop 2023-11-15T04:36:23.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 shutdown_connections 2023-11-15T04:36:23.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 --2- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe9bc02ff60 0x7fe9bc032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:23.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 --2- 172.21.15.190:0/1286904100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9cc0f5410 0x7fe9cc105630 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:23.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 >> 172.21.15.190:0/1286904100 conn(0x7fe9cc0f0b70 msgr2=0x7fe9cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:23.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 shutdown_connections 2023-11-15T04:36:23.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:23.604+0000 7fe9d2ee4700 1 -- 172.21.15.190:0/1286904100 wait complete. 2023-11-15T04:36:23.607 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:24.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:23.126884+0000 mgr.smithi047.zgysuz (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:24.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:23.602949+0000 mon.smithi047 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.190:0/1286904100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:24.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:24.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:26.160 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.159+0000 7fc16ff88700 1 -- 172.21.15.190:0/1860266469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 msgr2=0x7fc1680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:26.160 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.159+0000 7fc16ff88700 1 --2- 172.21.15.190:0/1860266469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 0x7fc1680f6930 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fc164009fb0 tx=0x7fc164009b30).stop 2023-11-15T04:36:26.160 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.159+0000 7fc16ff88700 1 -- 172.21.15.190:0/1860266469 shutdown_connections 2023-11-15T04:36:26.160 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.159+0000 7fc16ff88700 1 --2- 172.21.15.190:0/1860266469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 0x7fc1680f6930 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:26.161 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.159+0000 7fc16ff88700 1 -- 172.21.15.190:0/1860266469 >> 172.21.15.190:0/1860266469 conn(0x7fc1680f1c70 msgr2=0x7fc1680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:26.161 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.159+0000 7fc16ff88700 1 -- 172.21.15.190:0/1860266469 shutdown_connections 2023-11-15T04:36:26.161 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.159+0000 7fc16ff88700 1 -- 172.21.15.190:0/1860266469 wait complete. 2023-11-15T04:36:26.161 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.160+0000 7fc16ff88700 1 Processor -- start 2023-11-15T04:36:26.161 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.160+0000 7fc16ff88700 1 -- start start 2023-11-15T04:36:26.162 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.160+0000 7fc16ff88700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 0x7fc168106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:26.162 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.160+0000 7fc16ff88700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc168106c20 con 0x7fc1680f6510 2023-11-15T04:36:26.162 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.160+0000 7fc16dd24700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 0x7fc168106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:26.162 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.160+0000 7fc16dd24700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 0x7fc168106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:42448/0 (socket says 172.21.15.190:42448) 2023-11-15T04:36:26.162 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.160+0000 7fc16dd24700 1 -- 172.21.15.190:0/356806880 learned_addr learned my addr 172.21.15.190:0/356806880 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:26.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.160+0000 7fc16dd24700 1 -- 172.21.15.190:0/356806880 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164005040 con 0x7fc1680f6510 2023-11-15T04:36:26.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.160+0000 7fc16dd24700 1 --2- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 0x7fc168106730 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fc164013510 tx=0x7fc16400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:26.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.161+0000 7fc15e7fc700 1 -- 172.21.15.190:0/356806880 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc164009320 con 0x7fc1680f6510 2023-11-15T04:36:26.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.161+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc168106e20 con 0x7fc1680f6510 2023-11-15T04:36:26.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.161+0000 7fc15e7fc700 1 -- 172.21.15.190:0/356806880 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc164013f00 con 0x7fc1680f6510 2023-11-15T04:36:26.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.161+0000 7fc15e7fc700 1 -- 172.21.15.190:0/356806880 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc164014d90 con 0x7fc1680f6510 2023-11-15T04:36:26.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.161+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1681072b0 con 0x7fc1680f6510 2023-11-15T04:36:26.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.161+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc168045ad0 con 0x7fc1680f6510 2023-11-15T04:36:26.166 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.161+0000 7fc15e7fc700 1 -- 172.21.15.190:0/356806880 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc164014420 con 0x7fc1680f6510 2023-11-15T04:36:26.166 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.162+0000 7fc15e7fc700 1 --2- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc15402ff60 0x7fc154032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:26.166 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.162+0000 7fc15e7fc700 1 -- 172.21.15.190:0/356806880 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc164040ff0 con 0x7fc1680f6510 2023-11-15T04:36:26.166 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.162+0000 7fc16d523700 1 --2- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc15402ff60 0x7fc154032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:26.167 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.162+0000 7fc16d523700 1 --2- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc15402ff60 0x7fc154032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fc1580096a0 tx=0x7fc158006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:26.167 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.166+0000 7fc15e7fc700 1 -- 172.21.15.190:0/356806880 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1640095f0 con 0x7fc1680f6510 2023-11-15T04:36:26.373 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.374+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1680fbf60 con 0x7fc1680f6510 2023-11-15T04:36:26.374 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.375+0000 7fc15e7fc700 1 -- 172.21.15.190:0/356806880 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc16401db70 con 0x7fc1680f6510 2023-11-15T04:36:26.375 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:26.375 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:26.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc15402ff60 msgr2=0x7fc154032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:26.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 --2- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc15402ff60 0x7fc154032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fc1580096a0 tx=0x7fc158006b40).stop 2023-11-15T04:36:26.376 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 msgr2=0x7fc168106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:26.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 --2- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 0x7fc168106730 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fc164013510 tx=0x7fc16400b040).stop 2023-11-15T04:36:26.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 shutdown_connections 2023-11-15T04:36:26.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 --2- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc15402ff60 0x7fc154032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:26.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 --2- 172.21.15.190:0/356806880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1680f6510 0x7fc168106730 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:26.377 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 >> 172.21.15.190:0/356806880 conn(0x7fc1680f1c70 msgr2=0x7fc1680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:26.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 shutdown_connections 2023-11-15T04:36:26.378 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:26.376+0000 7fc16ff88700 1 -- 172.21.15.190:0/356806880 wait complete. 2023-11-15T04:36:26.378 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:26.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:25.127158+0000 mgr.smithi047.zgysuz (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:27.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:26.375063+0000 mon.smithi047 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.190:0/356806880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:27.750 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:27.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:28.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:28 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:27.127457+0000 mgr.smithi047.zgysuz (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:28.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.882+0000 7fa7ec03d700 1 -- 172.21.15.190:0/3633549308 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 msgr2=0x7fa7e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:28.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.882+0000 7fa7ec03d700 1 --2- 172.21.15.190:0/3633549308 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 0x7fa7e40f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fa7d4009fb0 tx=0x7fa7d4009b30).stop 2023-11-15T04:36:28.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.882+0000 7fa7ec03d700 1 -- 172.21.15.190:0/3633549308 shutdown_connections 2023-11-15T04:36:28.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.882+0000 7fa7ec03d700 1 --2- 172.21.15.190:0/3633549308 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 0x7fa7e40f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:28.884 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.882+0000 7fa7ec03d700 1 -- 172.21.15.190:0/3633549308 >> 172.21.15.190:0/3633549308 conn(0x7fa7e40f0b70 msgr2=0x7fa7e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:28.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.882+0000 7fa7ec03d700 1 -- 172.21.15.190:0/3633549308 shutdown_connections 2023-11-15T04:36:28.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.882+0000 7fa7ec03d700 1 -- 172.21.15.190:0/3633549308 wait complete. 2023-11-15T04:36:28.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.883+0000 7fa7ec03d700 1 Processor -- start 2023-11-15T04:36:28.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.883+0000 7fa7ec03d700 1 -- start start 2023-11-15T04:36:28.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.883+0000 7fa7ec03d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 0x7fa7e41055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:28.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.883+0000 7fa7ec03d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e4105ac0 con 0x7fa7e40f5410 2023-11-15T04:36:28.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.883+0000 7fa7e9dd9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 0x7fa7e41055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:28.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.883+0000 7fa7e9dd9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 0x7fa7e41055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:42470/0 (socket says 172.21.15.190:42470) 2023-11-15T04:36:28.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.883+0000 7fa7e9dd9700 1 -- 172.21.15.190:0/1059127929 learned_addr learned my addr 172.21.15.190:0/1059127929 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.883+0000 7fa7e9dd9700 1 -- 172.21.15.190:0/1059127929 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d4005040 con 0x7fa7e40f5410 2023-11-15T04:36:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.884+0000 7fa7e9dd9700 1 --2- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 0x7fa7e41055d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fa7d4013510 tx=0x7fa7d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.884+0000 7fa7da7fc700 1 -- 172.21.15.190:0/1059127929 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7d4009320 con 0x7fa7e40f5410 2023-11-15T04:36:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.884+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e4105cc0 con 0x7fa7e40f5410 2023-11-15T04:36:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.884+0000 7fa7da7fc700 1 -- 172.21.15.190:0/1059127929 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7d4013f00 con 0x7fa7e40f5410 2023-11-15T04:36:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.884+0000 7fa7da7fc700 1 -- 172.21.15.190:0/1059127929 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7d4014d90 con 0x7fa7e40f5410 2023-11-15T04:36:28.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.884+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e4106150 con 0x7fa7e40f5410 2023-11-15T04:36:28.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.885+0000 7fa7da7fc700 1 -- 172.21.15.190:0/1059127929 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7d40095f0 con 0x7fa7e40f5410 2023-11-15T04:36:28.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.885+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e4045ad0 con 0x7fa7e40f5410 2023-11-15T04:36:28.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.885+0000 7fa7da7fc700 1 --2- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7d002ff60 0x7fa7d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:28.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.885+0000 7fa7da7fc700 1 -- 172.21.15.190:0/1059127929 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7d4041000 con 0x7fa7e40f5410 2023-11-15T04:36:28.889 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.885+0000 7fa7e95d8700 1 --2- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7d002ff60 0x7fa7d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:28.890 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.885+0000 7fa7e95d8700 1 --2- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7d002ff60 0x7fa7d0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa7e00096a0 tx=0x7fa7e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:28.890 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:28.889+0000 7fa7da7fc700 1 -- 172.21.15.190:0/1059127929 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7d4014420 con 0x7fa7e40f5410 2023-11-15T04:36:29.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.096+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7e4107000 con 0x7fa7e40f5410 2023-11-15T04:36:29.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.097+0000 7fa7da7fc700 1 -- 172.21.15.190:0/1059127929 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa7d4010230 con 0x7fa7e40f5410 2023-11-15T04:36:29.097 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:29.097 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:29.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7d002ff60 msgr2=0x7fa7d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:29.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 --2- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7d002ff60 0x7fa7d0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa7e00096a0 tx=0x7fa7e0006b40).stop 2023-11-15T04:36:29.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 msgr2=0x7fa7e41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:29.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 --2- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 0x7fa7e41055d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fa7d4013510 tx=0x7fa7d400b040).stop 2023-11-15T04:36:29.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 shutdown_connections 2023-11-15T04:36:29.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 --2- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa7d002ff60 0x7fa7d0032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:29.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 --2- 172.21.15.190:0/1059127929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7e40f5410 0x7fa7e41055d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:29.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 >> 172.21.15.190:0/1059127929 conn(0x7fa7e40f0b70 msgr2=0x7fa7e40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:29.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 shutdown_connections 2023-11-15T04:36:29.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:29.098+0000 7fa7ec03d700 1 -- 172.21.15.190:0/1059127929 wait complete. 2023-11-15T04:36:29.101 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:29.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:29 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:29.097078+0000 mon.smithi047 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.190:0/1059127929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:30.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:30.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:30.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:29.127686+0000 mgr.smithi047.zgysuz (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:31.600 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.600+0000 7f0add933700 1 -- 172.21.15.190:0/102231701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 msgr2=0x7f0ad80f6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:31.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.600+0000 7f0add933700 1 --2- 172.21.15.190:0/102231701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 0x7f0ad80f6e00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f0ac8009fb0 tx=0x7f0ac8009b30).stop 2023-11-15T04:36:31.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.600+0000 7f0add933700 1 -- 172.21.15.190:0/102231701 shutdown_connections 2023-11-15T04:36:31.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.600+0000 7f0add933700 1 --2- 172.21.15.190:0/102231701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 0x7f0ad80f6e00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:31.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.600+0000 7f0add933700 1 -- 172.21.15.190:0/102231701 >> 172.21.15.190:0/102231701 conn(0x7f0ad80f02e0 msgr2=0x7f0ad80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:31.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.600+0000 7f0add933700 1 -- 172.21.15.190:0/102231701 shutdown_connections 2023-11-15T04:36:31.602 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.600+0000 7f0add933700 1 -- 172.21.15.190:0/102231701 wait complete. 2023-11-15T04:36:31.607 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.601+0000 7f0add933700 1 Processor -- start 2023-11-15T04:36:31.607 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.601+0000 7f0add933700 1 -- start start 2023-11-15T04:36:31.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.601+0000 7f0add933700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 0x7f0ad810b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:31.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.601+0000 7f0add933700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad8107200 con 0x7f0ad80f69e0 2023-11-15T04:36:31.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.601+0000 7f0ad6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 0x7f0ad810b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:31.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.601+0000 7f0ad6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 0x7f0ad810b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49480/0 (socket says 172.21.15.190:49480) 2023-11-15T04:36:31.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.602+0000 7f0ad6ffd700 1 -- 172.21.15.190:0/3977531479 learned_addr learned my addr 172.21.15.190:0/3977531479 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:31.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.602+0000 7f0ad6ffd700 1 -- 172.21.15.190:0/3977531479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac8005040 con 0x7f0ad80f69e0 2023-11-15T04:36:31.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.602+0000 7f0ad6ffd700 1 --2- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 0x7f0ad810b810 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f0ac8013510 tx=0x7f0ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:31.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.602+0000 7f0abf7fe700 1 -- 172.21.15.190:0/3977531479 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac8009320 con 0x7f0ad80f69e0 2023-11-15T04:36:31.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.602+0000 7f0abf7fe700 1 -- 172.21.15.190:0/3977531479 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ac8013f00 con 0x7f0ad80f69e0 2023-11-15T04:36:31.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.602+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad80f5350 con 0x7f0ad80f69e0 2023-11-15T04:36:31.613 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.602+0000 7f0abf7fe700 1 -- 172.21.15.190:0/3977531479 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac8014d90 con 0x7f0ad80f69e0 2023-11-15T04:36:31.613 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.602+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad810bd00 con 0x7f0ad80f69e0 2023-11-15T04:36:31.613 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.603+0000 7f0abf7fe700 1 -- 172.21.15.190:0/3977531479 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ac80095f0 con 0x7f0ad80f69e0 2023-11-15T04:36:31.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.603+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad8045ad0 con 0x7f0ad80f69e0 2023-11-15T04:36:31.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.603+0000 7f0abf7fe700 1 --2- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0ac002ff60 0x7f0ac0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:31.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.603+0000 7f0abf7fe700 1 -- 172.21.15.190:0/3977531479 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ac8041000 con 0x7f0ad80f69e0 2023-11-15T04:36:31.614 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.603+0000 7f0ad67fc700 1 --2- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0ac002ff60 0x7f0ac0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:31.615 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.604+0000 7f0ad67fc700 1 --2- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0ac002ff60 0x7f0ac0032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f0acc0096a0 tx=0x7f0acc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:31.616 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.608+0000 7f0abf7fe700 1 -- 172.21.15.190:0/3977531479 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ac801d7f0 con 0x7f0ad80f69e0 2023-11-15T04:36:31.814 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.815+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ad810c4c0 con 0x7f0ad80f69e0 2023-11-15T04:36:31.815 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.815+0000 7f0abf7fe700 1 -- 172.21.15.190:0/3977531479 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ac8010230 con 0x7f0ad80f69e0 2023-11-15T04:36:31.816 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:31.816 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:31.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0ac002ff60 msgr2=0x7f0ac0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:31.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 --2- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0ac002ff60 0x7f0ac0032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f0acc0096a0 tx=0x7f0acc006b40).stop 2023-11-15T04:36:31.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 msgr2=0x7f0ad810b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:31.817 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 --2- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 0x7f0ad810b810 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f0ac8013510 tx=0x7f0ac800b040).stop 2023-11-15T04:36:31.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 shutdown_connections 2023-11-15T04:36:31.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 --2- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0ac002ff60 0x7f0ac0032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:31.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 --2- 172.21.15.190:0/3977531479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad80f69e0 0x7f0ad810b810 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:31.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 >> 172.21.15.190:0/3977531479 conn(0x7f0ad80f02e0 msgr2=0x7f0ad80f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:31.818 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 shutdown_connections 2023-11-15T04:36:31.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:31.817+0000 7f0add933700 1 -- 172.21.15.190:0/3977531479 wait complete. 2023-11-15T04:36:31.820 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:31.127935+0000 mgr.smithi047.zgysuz (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:31.815823+0000 mon.smithi047 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.190:0/3977531479' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:33.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:33.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:34.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.341+0000 7fc28d425700 1 -- 172.21.15.190:0/3333619860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 msgr2=0x7fc2880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:34.343 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.341+0000 7fc28d425700 1 --2- 172.21.15.190:0/3333619860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 0x7fc2880f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fc270009fb0 tx=0x7fc270009b30).stop 2023-11-15T04:36:34.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.341+0000 7fc28d425700 1 -- 172.21.15.190:0/3333619860 shutdown_connections 2023-11-15T04:36:34.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.341+0000 7fc28d425700 1 --2- 172.21.15.190:0/3333619860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 0x7fc2880f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:34.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.341+0000 7fc28d425700 1 -- 172.21.15.190:0/3333619860 >> 172.21.15.190:0/3333619860 conn(0x7fc2880f0b70 msgr2=0x7fc2880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:34.344 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.341+0000 7fc28d425700 1 -- 172.21.15.190:0/3333619860 shutdown_connections 2023-11-15T04:36:34.345 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.341+0000 7fc28d425700 1 -- 172.21.15.190:0/3333619860 wait complete. 2023-11-15T04:36:34.345 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.342+0000 7fc28d425700 1 Processor -- start 2023-11-15T04:36:34.345 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.342+0000 7fc28d425700 1 -- start start 2023-11-15T04:36:34.345 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.342+0000 7fc28d425700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 0x7fc2881066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:34.346 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.342+0000 7fc28d425700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc288106bc0 con 0x7fc2880f5410 2023-11-15T04:36:34.346 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.342+0000 7fc286ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 0x7fc2881066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:34.346 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.342+0000 7fc286ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 0x7fc2881066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49502/0 (socket says 172.21.15.190:49502) 2023-11-15T04:36:34.347 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.342+0000 7fc286ffd700 1 -- 172.21.15.190:0/2366149682 learned_addr learned my addr 172.21.15.190:0/2366149682 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:34.347 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.342+0000 7fc286ffd700 1 -- 172.21.15.190:0/2366149682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270005040 con 0x7fc2880f5410 2023-11-15T04:36:34.347 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.343+0000 7fc286ffd700 1 --2- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 0x7fc2881066d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fc270014000 tx=0x7fc27000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:34.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.343+0000 7fc27f7fe700 1 -- 172.21.15.190:0/2366149682 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2700095f0 con 0x7fc2880f5410 2023-11-15T04:36:34.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.343+0000 7fc27f7fe700 1 -- 172.21.15.190:0/2366149682 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc270007e70 con 0x7fc2880f5410 2023-11-15T04:36:34.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.343+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc288106dc0 con 0x7fc2880f5410 2023-11-15T04:36:34.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.343+0000 7fc27f7fe700 1 -- 172.21.15.190:0/2366149682 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc270014ea0 con 0x7fc2880f5410 2023-11-15T04:36:34.349 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.343+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc288107250 con 0x7fc2880f5410 2023-11-15T04:36:34.350 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.344+0000 7fc27f7fe700 1 -- 172.21.15.190:0/2366149682 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc270025070 con 0x7fc2880f5410 2023-11-15T04:36:34.351 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.344+0000 7fc27f7fe700 1 --2- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc27402ff60 0x7fc274032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:34.351 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.344+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc288045ad0 con 0x7fc2880f5410 2023-11-15T04:36:34.351 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.344+0000 7fc27f7fe700 1 -- 172.21.15.190:0/2366149682 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc270040a30 con 0x7fc2880f5410 2023-11-15T04:36:34.351 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.344+0000 7fc2867fc700 1 --2- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc27402ff60 0x7fc274032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:34.352 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.345+0000 7fc2867fc700 1 --2- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc27402ff60 0x7fc274032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc2780096a0 tx=0x7fc278006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:34.352 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.349+0000 7fc27f7fe700 1 -- 172.21.15.190:0/2366149682 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2700079c0 con 0x7fc2880f5410 2023-11-15T04:36:34.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:33.128176+0000 mgr.smithi047.zgysuz (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:34.555 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.555+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc288108100 con 0x7fc2880f5410 2023-11-15T04:36:34.557 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.556+0000 7fc27f7fe700 1 -- 172.21.15.190:0/2366149682 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc27000f300 con 0x7fc2880f5410 2023-11-15T04:36:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:34.558 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:34.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc27402ff60 msgr2=0x7fc274032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:34.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 --2- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc27402ff60 0x7fc274032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc2780096a0 tx=0x7fc278006b40).stop 2023-11-15T04:36:34.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 msgr2=0x7fc2881066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:34.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 --2- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 0x7fc2881066d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fc270014000 tx=0x7fc27000b040).stop 2023-11-15T04:36:34.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 shutdown_connections 2023-11-15T04:36:34.562 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 --2- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fc27402ff60 0x7fc274032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:34.562 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 --2- 172.21.15.190:0/2366149682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2880f5410 0x7fc2881066d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:34.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 >> 172.21.15.190:0/2366149682 conn(0x7fc2880f0b70 msgr2=0x7fc28805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:34.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 shutdown_connections 2023-11-15T04:36:34.564 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:34.558+0000 7fc28d425700 1 -- 172.21.15.190:0/2366149682 wait complete. 2023-11-15T04:36:34.564 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:35.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:35 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:34.556707+0000 mon.smithi047 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.190:0/2366149682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:35.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:35.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:35.128416+0000 mgr.smithi047.zgysuz (mgr.14164) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:37.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.195+0000 7f7404ef7700 1 -- 172.21.15.190:0/1532450342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 msgr2=0x7f74000fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:37.196 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.195+0000 7f7404ef7700 1 --2- 172.21.15.190:0/1532450342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 0x7f74000fb570 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f73e8009fb0 tx=0x7f73e8009b30).stop 2023-11-15T04:36:37.196 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.195+0000 7f7404ef7700 1 -- 172.21.15.190:0/1532450342 shutdown_connections 2023-11-15T04:36:37.196 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.195+0000 7f7404ef7700 1 --2- 172.21.15.190:0/1532450342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 0x7f74000fb570 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:37.196 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.195+0000 7f7404ef7700 1 -- 172.21.15.190:0/1532450342 >> 172.21.15.190:0/1532450342 conn(0x7f74000f0b70 msgr2=0x7f74000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:37.196 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.195+0000 7f7404ef7700 1 -- 172.21.15.190:0/1532450342 shutdown_connections 2023-11-15T04:36:37.197 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.195+0000 7f7404ef7700 1 -- 172.21.15.190:0/1532450342 wait complete. 2023-11-15T04:36:37.199 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.196+0000 7f7404ef7700 1 Processor -- start 2023-11-15T04:36:37.199 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.196+0000 7f7404ef7700 1 -- start start 2023-11-15T04:36:37.199 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.196+0000 7f7404ef7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 0x7f74001088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:37.199 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.196+0000 7f7404ef7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7400108dc0 con 0x7f7400060200 2023-11-15T04:36:37.199 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.196+0000 7f73fe59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 0x7f74001088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.196+0000 7f73fe59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 0x7f74001088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49516/0 (socket says 172.21.15.190:49516) 2023-11-15T04:36:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.196+0000 7f73fe59c700 1 -- 172.21.15.190:0/221492150 learned_addr learned my addr 172.21.15.190:0/221492150 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.197+0000 7f73fe59c700 1 -- 172.21.15.190:0/221492150 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8005040 con 0x7f7400060200 2023-11-15T04:36:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.197+0000 7f73fe59c700 1 --2- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 0x7f74001088d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f73e8009830 tx=0x7f73e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.197+0000 7f73f6ffd700 1 -- 172.21.15.190:0/221492150 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73e8009320 con 0x7f7400060200 2023-11-15T04:36:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.197+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7400108fc0 con 0x7f7400060200 2023-11-15T04:36:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.197+0000 7f73f6ffd700 1 -- 172.21.15.190:0/221492150 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73e80136c0 con 0x7f7400060200 2023-11-15T04:36:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.197+0000 7f73f6ffd700 1 -- 172.21.15.190:0/221492150 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73e801ee90 con 0x7f7400060200 2023-11-15T04:36:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.197+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7400109450 con 0x7f7400060200 2023-11-15T04:36:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.198+0000 7f73f6ffd700 1 -- 172.21.15.190:0/221492150 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73e801e420 con 0x7f7400060200 2023-11-15T04:36:37.202 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.198+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7400045ad0 con 0x7f7400060200 2023-11-15T04:36:37.202 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.198+0000 7f73f6ffd700 1 --2- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73ec02ff10 0x7f73ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:37.202 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.198+0000 7f73f6ffd700 1 -- 172.21.15.190:0/221492150 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f73e8040f40 con 0x7f7400060200 2023-11-15T04:36:37.202 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.198+0000 7f73fdd9b700 1 --2- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73ec02ff10 0x7f73ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:37.202 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.199+0000 7f73fdd9b700 1 --2- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73ec02ff10 0x7f73ec0323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f73f00096a0 tx=0x7f73f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:37.203 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.203+0000 7f73f6ffd700 1 -- 172.21.15.190:0/221492150 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73e806d2b0 con 0x7f7400060200 2023-11-15T04:36:37.406 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.406+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f740010a300 con 0x7f7400060200 2023-11-15T04:36:37.407 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.407+0000 7f73f6ffd700 1 -- 172.21.15.190:0/221492150 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f73e80095f0 con 0x7f7400060200 2023-11-15T04:36:37.407 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:37.408 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:37.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.408+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73ec02ff10 msgr2=0x7f73ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:37.409 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.408+0000 7f7404ef7700 1 --2- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73ec02ff10 0x7f73ec0323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f73f00096a0 tx=0x7f73f0006b40).stop 2023-11-15T04:36:37.409 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.408+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 msgr2=0x7f74001088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:37.409 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.408+0000 7f7404ef7700 1 --2- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 0x7f74001088d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f73e8009830 tx=0x7f73e800b040).stop 2023-11-15T04:36:37.409 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.408+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 shutdown_connections 2023-11-15T04:36:37.409 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.408+0000 7f7404ef7700 1 --2- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73ec02ff10 0x7f73ec0323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:37.410 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.408+0000 7f7404ef7700 1 --2- 172.21.15.190:0/221492150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7400060200 0x7f74001088d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:37.410 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.408+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 >> 172.21.15.190:0/221492150 conn(0x7f74000f0b70 msgr2=0x7f74000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:37.410 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.409+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 shutdown_connections 2023-11-15T04:36:37.410 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:37.409+0000 7f7404ef7700 1 -- 172.21.15.190:0/221492150 wait complete. 2023-11-15T04:36:37.411 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:37.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:36:35.799586+0000 mgr.smithi047.zgysuz (mgr.14164) 232 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi047 ... 2023-11-15T04:36:37.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Verifying port 9100 ... 2023-11-15T04:36:37.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Non-zero exit code 1 from systemctl start ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@node-exporter.smithi047 2023-11-15T04:36:37.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: systemctl: stderr Job for ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@node-exporter.smithi047.service failed because a timeout was exceeded. 2023-11-15T04:36:37.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: systemctl: stderr See "systemctl status ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@node-exporter.smithi047.service" and "journalctl -xe" for details. 2023-11-15T04:36:37.509 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Traceback (most recent call last): 2023-11-15T04:36:37.509 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T04:36:37.509 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: main() 2023-11-15T04:36:37.509 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T04:36:37.509 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: r = ctx.func(ctx) 2023-11-15T04:36:37.510 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T04:36:37.510 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: return func(ctx) 2023-11-15T04:36:37.510 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T04:36:37.510 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ports=daemon_ports) 2023-11-15T04:36:37.510 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T04:36:37.511 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T04:36:37.511 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T04:36:37.511 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T04:36:37.511 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T04:36:37.511 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T04:36:37.511 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: RuntimeError: Failed command: systemctl start ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@node-exporter.smithi047 2023-11-15T04:36:37.512 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Traceback (most recent call last): 2023-11-15T04:36:37.512 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T04:36:37.512 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: yield (conn, connr) 2023-11-15T04:36:37.512 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T04:36:37.512 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: code, '\n'.join(err))) 2023-11-15T04:36:37.513 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi047 ... 2023-11-15T04:36:37.513 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Verifying port 9100 ... 2023-11-15T04:36:37.513 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Non-zero exit code 1 from systemctl start ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@node-exporter.smithi047 2023-11-15T04:36:37.513 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: systemctl: stderr Job for ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@node-exporter.smithi047.service failed because a timeout was exceeded. 2023-11-15T04:36:37.513 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: systemctl: stderr See "systemctl status ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@node-exporter.smithi047.service" and "journalctl -xe" for details. 2023-11-15T04:36:37.514 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: Traceback (most recent call last): 2023-11-15T04:36:37.514 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T04:36:37.514 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: main() 2023-11-15T04:36:37.514 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T04:36:37.514 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: r = ctx.func(ctx) 2023-11-15T04:36:37.515 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T04:36:37.515 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: return func(ctx) 2023-11-15T04:36:37.515 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T04:36:37.515 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: ports=daemon_ports) 2023-11-15T04:36:37.515 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T04:36:37.515 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T04:36:37.516 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T04:36:37.517 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T04:36:37.517 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: File "/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T04:36:37.517 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T04:36:37.517 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: RuntimeError: Failed command: systemctl start ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@node-exporter.smithi047 2023-11-15T04:36:37.518 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cephadm 2023-11-15T04:36:35.806942+0000 mgr.smithi047.zgysuz (mgr.14164) 233 : cephadm [INF] Deploying daemon prometheus.smithi047 on smithi047 2023-11-15T04:36:38.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:38 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:37.128784+0000 mgr.smithi047.zgysuz (mgr.14164) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:38.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:38 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:37.407538+0000 mon.smithi047 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.190:0/221492150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:38.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:38.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:39.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.966+0000 7f7c42177700 1 -- 172.21.15.190:0/3414750225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c0f6510 msgr2=0x7f7c3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:39.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.966+0000 7f7c42177700 1 --2- 172.21.15.190:0/3414750225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c0f6510 0x7f7c3c0f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f7c2c009fb0 tx=0x7f7c2c009b30).stop 2023-11-15T04:36:39.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.966+0000 7f7c42177700 1 -- 172.21.15.190:0/3414750225 shutdown_connections 2023-11-15T04:36:39.968 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.966+0000 7f7c42177700 1 --2- 172.21.15.190:0/3414750225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c0f6510 0x7f7c3c0f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:39.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.966+0000 7f7c42177700 1 -- 172.21.15.190:0/3414750225 >> 172.21.15.190:0/3414750225 conn(0x7f7c3c0f1c70 msgr2=0x7f7c3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:39.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.966+0000 7f7c42177700 1 -- 172.21.15.190:0/3414750225 shutdown_connections 2023-11-15T04:36:39.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.966+0000 7f7c42177700 1 -- 172.21.15.190:0/3414750225 wait complete. 2023-11-15T04:36:39.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.967+0000 7f7c42177700 1 Processor -- start 2023-11-15T04:36:39.969 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.967+0000 7f7c42177700 1 -- start start 2023-11-15T04:36:39.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.967+0000 7f7c42177700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c108a20 0x7f7c3c108e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:39.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.967+0000 7f7c42177700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c109330 con 0x7f7c3c108a20 2023-11-15T04:36:39.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.967+0000 7f7c3b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c108a20 0x7f7c3c108e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:39.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.967+0000 7f7c3b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c108a20 0x7f7c3c108e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:49538/0 (socket says 172.21.15.190:49538) 2023-11-15T04:36:39.970 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.967+0000 7f7c3b7fe700 1 -- 172.21.15.190:0/999603314 learned_addr learned my addr 172.21.15.190:0/999603314 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:39.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.967+0000 7f7c3b7fe700 1 -- 172.21.15.190:0/999603314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c005040 con 0x7f7c3c108a20 2023-11-15T04:36:39.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.968+0000 7f7c3b7fe700 1 --2- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c108a20 0x7f7c3c108e40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f7c2c009f80 tx=0x7f7c2c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:39.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.968+0000 7f7c33fff700 1 -- 172.21.15.190:0/999603314 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c2c0136c0 con 0x7f7c3c108a20 2023-11-15T04:36:39.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.968+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c109530 con 0x7f7c3c108a20 2023-11-15T04:36:39.971 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.968+0000 7f7c33fff700 1 -- 172.21.15.190:0/999603314 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c2c013e70 con 0x7f7c3c108a20 2023-11-15T04:36:39.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.968+0000 7f7c33fff700 1 -- 172.21.15.190:0/999603314 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c2c014420 con 0x7f7c3c108a20 2023-11-15T04:36:39.972 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.968+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c10a080 con 0x7f7c3c108a20 2023-11-15T04:36:39.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.969+0000 7f7c33fff700 1 -- 172.21.15.190:0/999603314 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c2c025060 con 0x7f7c3c108a20 2023-11-15T04:36:39.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.969+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c059100 con 0x7f7c3c108a20 2023-11-15T04:36:39.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.969+0000 7f7c33fff700 1 --2- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c2802ff60 0x7f7c28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:39.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.969+0000 7f7c33fff700 1 -- 172.21.15.190:0/999603314 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c2c040f10 con 0x7f7c3c108a20 2023-11-15T04:36:39.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.969+0000 7f7c3affd700 1 --2- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c2802ff60 0x7f7c28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:39.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.969+0000 7f7c3affd700 1 --2- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c2802ff60 0x7f7c28032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f7c240096a0 tx=0x7f7c24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:39.975 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:39.974+0000 7f7c33fff700 1 -- 172.21.15.190:0/999603314 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c2c01cb30 con 0x7f7c3c108a20 2023-11-15T04:36:40.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.179+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c3c045ad0 con 0x7f7c3c108a20 2023-11-15T04:36:40.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.180+0000 7f7c33fff700 1 -- 172.21.15.190:0/999603314 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c2c02d2d0 con 0x7f7c3c108a20 2023-11-15T04:36:40.180 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:40.180 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:40.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c2802ff60 msgr2=0x7f7c28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:40.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 --2- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c2802ff60 0x7f7c28032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f7c240096a0 tx=0x7f7c24006b40).stop 2023-11-15T04:36:40.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c108a20 msgr2=0x7f7c3c108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:40.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 --2- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c108a20 0x7f7c3c108e40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f7c2c009f80 tx=0x7f7c2c009b30).stop 2023-11-15T04:36:40.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 shutdown_connections 2023-11-15T04:36:40.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 --2- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7c2802ff60 0x7f7c28032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:40.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 --2- 172.21.15.190:0/999603314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c3c108a20 0x7f7c3c108e40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:40.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 >> 172.21.15.190:0/999603314 conn(0x7f7c3c0f1c70 msgr2=0x7f7c3c0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:40.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 shutdown_connections 2023-11-15T04:36:40.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:40.181+0000 7f7c42177700 1 -- 172.21.15.190:0/999603314 wait complete. 2023-11-15T04:36:40.183 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:40.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:39.129184+0000 mgr.smithi047.zgysuz (mgr.14164) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:40.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:40.180153+0000 mon.smithi047 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.190:0/999603314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:41.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:41.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:42.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:42 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:41.129544+0000 mgr.smithi047.zgysuz (mgr.14164) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:42.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.779+0000 7f5941b0f700 1 -- 172.21.15.190:0/3205859786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 msgr2=0x7f593c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:42.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.779+0000 7f5941b0f700 1 --2- 172.21.15.190:0/3205859786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 0x7f593c0f7d00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f5934009fb0 tx=0x7f5934009b30).stop 2023-11-15T04:36:42.780 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.779+0000 7f5941b0f700 1 -- 172.21.15.190:0/3205859786 shutdown_connections 2023-11-15T04:36:42.781 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.779+0000 7f5941b0f700 1 --2- 172.21.15.190:0/3205859786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 0x7f593c0f7d00 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:42.781 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.779+0000 7f5941b0f700 1 -- 172.21.15.190:0/3205859786 >> 172.21.15.190:0/3205859786 conn(0x7f593c0f0120 msgr2=0x7f593c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:42.781 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.779+0000 7f5941b0f700 1 -- 172.21.15.190:0/3205859786 shutdown_connections 2023-11-15T04:36:42.781 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.779+0000 7f5941b0f700 1 -- 172.21.15.190:0/3205859786 wait complete. 2023-11-15T04:36:42.781 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.779+0000 7f5941b0f700 1 Processor -- start 2023-11-15T04:36:42.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.779+0000 7f5941b0f700 1 -- start start 2023-11-15T04:36:42.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f5941b0f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 0x7f593c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:42.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f5941b0f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c1069d0 con 0x7f593c0f78e0 2023-11-15T04:36:42.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f5940b0d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 0x7f593c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:42.782 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f5940b0d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 0x7f593c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41262/0 (socket says 172.21.15.190:41262) 2023-11-15T04:36:42.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f5940b0d700 1 -- 172.21.15.190:0/2675365854 learned_addr learned my addr 172.21.15.190:0/2675365854 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:42.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f5940b0d700 1 -- 172.21.15.190:0/2675365854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5934005040 con 0x7f593c0f78e0 2023-11-15T04:36:42.783 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f5940b0d700 1 --2- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 0x7f593c1064e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f5934013510 tx=0x7f593400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:42.784 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f59397fa700 1 -- 172.21.15.190:0/2675365854 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5934009320 con 0x7f593c0f78e0 2023-11-15T04:36:42.784 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c106bd0 con 0x7f593c0f78e0 2023-11-15T04:36:42.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f59397fa700 1 -- 172.21.15.190:0/2675365854 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5934013f00 con 0x7f593c0f78e0 2023-11-15T04:36:42.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.780+0000 7f59397fa700 1 -- 172.21.15.190:0/2675365854 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5934014dc0 con 0x7f593c0f78e0 2023-11-15T04:36:42.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.781+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c107060 con 0x7f593c0f78e0 2023-11-15T04:36:42.785 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.781+0000 7f59397fa700 1 -- 172.21.15.190:0/2675365854 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59340095f0 con 0x7f593c0f78e0 2023-11-15T04:36:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.781+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c045ad0 con 0x7f593c0f78e0 2023-11-15T04:36:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.781+0000 7f59397fa700 1 --2- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f592802ff60 0x7f5928032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.781+0000 7f59397fa700 1 -- 172.21.15.190:0/2675365854 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5934041040 con 0x7f593c0f78e0 2023-11-15T04:36:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.782+0000 7f593bfff700 1 --2- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f592802ff60 0x7f5928032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.782+0000 7f593bfff700 1 --2- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f592802ff60 0x7f5928032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f59240096a0 tx=0x7f5924006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.786+0000 7f59397fa700 1 -- 172.21.15.190:0/2675365854 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5934014420 con 0x7f593c0f78e0 2023-11-15T04:36:42.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.993+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f593c107f10 con 0x7f593c0f78e0 2023-11-15T04:36:42.993 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.994+0000 7f59397fa700 1 -- 172.21.15.190:0/2675365854 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5934010230 con 0x7f593c0f78e0 2023-11-15T04:36:42.994 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:42.994 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:42.995 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f592802ff60 msgr2=0x7f5928032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:42.995 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 --2- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f592802ff60 0x7f5928032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f59240096a0 tx=0x7f5924006b40).stop 2023-11-15T04:36:42.996 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 msgr2=0x7f593c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:42.996 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 --2- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 0x7f593c1064e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f5934013510 tx=0x7f593400b040).stop 2023-11-15T04:36:42.996 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 shutdown_connections 2023-11-15T04:36:42.996 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 --2- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f592802ff60 0x7f5928032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:42.996 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 --2- 172.21.15.190:0/2675365854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c0f78e0 0x7f593c1064e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:42.997 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 >> 172.21.15.190:0/2675365854 conn(0x7f593c0f0120 msgr2=0x7f593c0f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:42.997 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 shutdown_connections 2023-11-15T04:36:42.997 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:42.995+0000 7f5941b0f700 1 -- 172.21.15.190:0/2675365854 wait complete. 2023-11-15T04:36:42.997 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:43.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:43 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:42.994173+0000 mon.smithi047 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.190:0/2675365854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:44.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:44.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:43.129942+0000 mgr.smithi047.zgysuz (mgr.14164) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:45.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.557+0000 7f16527f7700 1 -- 172.21.15.190:0/898580838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 msgr2=0x7f164c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:45.558 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.557+0000 7f16527f7700 1 --2- 172.21.15.190:0/898580838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 0x7f164c0f6930 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f163c009fb0 tx=0x7f163c009b30).stop 2023-11-15T04:36:45.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.557+0000 7f16527f7700 1 -- 172.21.15.190:0/898580838 shutdown_connections 2023-11-15T04:36:45.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.557+0000 7f16527f7700 1 --2- 172.21.15.190:0/898580838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 0x7f164c0f6930 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:45.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.557+0000 7f16527f7700 1 -- 172.21.15.190:0/898580838 >> 172.21.15.190:0/898580838 conn(0x7f164c0f1c70 msgr2=0x7f164c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:45.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.557+0000 7f16527f7700 1 -- 172.21.15.190:0/898580838 shutdown_connections 2023-11-15T04:36:45.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.557+0000 7f16527f7700 1 -- 172.21.15.190:0/898580838 wait complete. 2023-11-15T04:36:45.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.558+0000 7f16527f7700 1 Processor -- start 2023-11-15T04:36:45.563 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.558+0000 7f16527f7700 1 -- start start 2023-11-15T04:36:45.564 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.558+0000 7f16527f7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 0x7f164c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:45.564 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.558+0000 7f16527f7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c106c20 con 0x7f164c0f6510 2023-11-15T04:36:45.564 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.558+0000 7f164bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 0x7f164c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:45.564 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.558+0000 7f164bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 0x7f164c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41278/0 (socket says 172.21.15.190:41278) 2023-11-15T04:36:45.565 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.558+0000 7f164bfff700 1 -- 172.21.15.190:0/2799490639 learned_addr learned my addr 172.21.15.190:0/2799490639 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:45.565 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.559+0000 7f164bfff700 1 -- 172.21.15.190:0/2799490639 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c005040 con 0x7f164c0f6510 2023-11-15T04:36:45.565 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.559+0000 7f164bfff700 1 --2- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 0x7f164c106730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f163c013510 tx=0x7f163c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:45.565 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.559+0000 7f1648ff9700 1 -- 172.21.15.190:0/2799490639 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f163c009320 con 0x7f164c0f6510 2023-11-15T04:36:45.565 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.559+0000 7f1648ff9700 1 -- 172.21.15.190:0/2799490639 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f163c013f00 con 0x7f164c0f6510 2023-11-15T04:36:45.566 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.559+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164c106e20 con 0x7f164c0f6510 2023-11-15T04:36:45.566 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.559+0000 7f1648ff9700 1 -- 172.21.15.190:0/2799490639 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f163c014d90 con 0x7f164c0f6510 2023-11-15T04:36:45.566 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.559+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164c1072b0 con 0x7f164c0f6510 2023-11-15T04:36:45.566 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.560+0000 7f1648ff9700 1 -- 172.21.15.190:0/2799490639 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f163c0095f0 con 0x7f164c0f6510 2023-11-15T04:36:45.567 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.560+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164c045ad0 con 0x7f164c0f6510 2023-11-15T04:36:45.567 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.560+0000 7f1648ff9700 1 --2- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f163402ff60 0x7f1634032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:45.567 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.560+0000 7f1648ff9700 1 -- 172.21.15.190:0/2799490639 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f163c040fe0 con 0x7f164c0f6510 2023-11-15T04:36:45.567 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.560+0000 7f164b7fe700 1 --2- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f163402ff60 0x7f1634032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:45.567 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.561+0000 7f164b7fe700 1 --2- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f163402ff60 0x7f1634032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f16400096a0 tx=0x7f1640006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:45.568 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.565+0000 7f1648ff9700 1 -- 172.21.15.190:0/2799490639 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f163c01d7f0 con 0x7f164c0f6510 2023-11-15T04:36:45.770 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.770+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f164c0fbf60 con 0x7f164c0f6510 2023-11-15T04:36:45.771 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.770+0000 7f1648ff9700 1 -- 172.21.15.190:0/2799490639 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f163c010230 con 0x7f164c0f6510 2023-11-15T04:36:45.771 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:45.772 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:45.773 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f163402ff60 msgr2=0x7f1634032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:45.773 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 --2- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f163402ff60 0x7f1634032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f16400096a0 tx=0x7f1640006b40).stop 2023-11-15T04:36:45.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 msgr2=0x7f164c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:45.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 --2- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 0x7f164c106730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f163c013510 tx=0x7f163c00b040).stop 2023-11-15T04:36:45.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 shutdown_connections 2023-11-15T04:36:45.774 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 --2- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f163402ff60 0x7f1634032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:45.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 --2- 172.21.15.190:0/2799490639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f164c0f6510 0x7f164c106730 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:45.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 >> 172.21.15.190:0/2799490639 conn(0x7f164c0f1c70 msgr2=0x7f164c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:45.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 shutdown_connections 2023-11-15T04:36:45.775 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:45.772+0000 7f16527f7700 1 -- 172.21.15.190:0/2799490639 wait complete. 2023-11-15T04:36:45.776 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:45.130302+0000 mgr.smithi047.zgysuz (mgr.14164) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:45.771945+0000 mon.smithi047 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.190:0/2799490639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:47.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:47.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:48.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:47.130681+0000 mgr.smithi047.zgysuz (mgr.14164) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:48.288 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.287+0000 7f4e34cfc700 1 -- 172.21.15.190:0/4210502710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 msgr2=0x7f4e300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:48.289 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.287+0000 7f4e34cfc700 1 --2- 172.21.15.190:0/4210502710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 0x7f4e300f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f4e20009fb0 tx=0x7f4e20009b30).stop 2023-11-15T04:36:48.289 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.287+0000 7f4e34cfc700 1 -- 172.21.15.190:0/4210502710 shutdown_connections 2023-11-15T04:36:48.289 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.287+0000 7f4e34cfc700 1 --2- 172.21.15.190:0/4210502710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 0x7f4e300f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:48.289 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.287+0000 7f4e34cfc700 1 -- 172.21.15.190:0/4210502710 >> 172.21.15.190:0/4210502710 conn(0x7f4e300f0b70 msgr2=0x7f4e300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:48.290 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.287+0000 7f4e34cfc700 1 -- 172.21.15.190:0/4210502710 shutdown_connections 2023-11-15T04:36:48.290 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.287+0000 7f4e34cfc700 1 -- 172.21.15.190:0/4210502710 wait complete. 2023-11-15T04:36:48.292 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.288+0000 7f4e34cfc700 1 Processor -- start 2023-11-15T04:36:48.292 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.288+0000 7f4e34cfc700 1 -- start start 2023-11-15T04:36:48.292 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.288+0000 7f4e34cfc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 0x7f4e30105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:48.293 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.288+0000 7f4e34cfc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e30105b20 con 0x7f4e300f5410 2023-11-15T04:36:48.293 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e2e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 0x7f4e30105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:48.293 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e2e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 0x7f4e30105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:41288/0 (socket says 172.21.15.190:41288) 2023-11-15T04:36:48.293 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e2e59c700 1 -- 172.21.15.190:0/416142028 learned_addr learned my addr 172.21.15.190:0/416142028 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:48.293 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e2e59c700 1 -- 172.21.15.190:0/416142028 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e20005040 con 0x7f4e300f5410 2023-11-15T04:36:48.294 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e2e59c700 1 --2- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 0x7f4e30105630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f4e20013510 tx=0x7f4e2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:48.294 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e26ffd700 1 -- 172.21.15.190:0/416142028 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e20009320 con 0x7f4e300f5410 2023-11-15T04:36:48.294 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e30105d20 con 0x7f4e300f5410 2023-11-15T04:36:48.294 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e26ffd700 1 -- 172.21.15.190:0/416142028 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e20013f00 con 0x7f4e300f5410 2023-11-15T04:36:48.295 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e26ffd700 1 -- 172.21.15.190:0/416142028 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e20014d90 con 0x7f4e300f5410 2023-11-15T04:36:48.296 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.289+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e301061b0 con 0x7f4e300f5410 2023-11-15T04:36:48.296 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.290+0000 7f4e26ffd700 1 -- 172.21.15.190:0/416142028 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e20014420 con 0x7f4e300f5410 2023-11-15T04:36:48.296 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.290+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e30045ad0 con 0x7f4e300f5410 2023-11-15T04:36:48.296 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.290+0000 7f4e26ffd700 1 --2- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4e1c02ff60 0x7f4e1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:48.297 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.291+0000 7f4e26ffd700 1 -- 172.21.15.190:0/416142028 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e20041000 con 0x7f4e300f5410 2023-11-15T04:36:48.297 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.291+0000 7f4e2dd9b700 1 --2- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4e1c02ff60 0x7f4e1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:48.297 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.291+0000 7f4e2dd9b700 1 --2- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4e1c02ff60 0x7f4e1c032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f4e180096a0 tx=0x7f4e18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:48.297 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.295+0000 7f4e26ffd700 1 -- 172.21.15.190:0/416142028 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e200095f0 con 0x7f4e300f5410 2023-11-15T04:36:48.505 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.505+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e300fae60 con 0x7f4e300f5410 2023-11-15T04:36:48.506 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.505+0000 7f4e26ffd700 1 -- 172.21.15.190:0/416142028 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e2001db70 con 0x7f4e300f5410 2023-11-15T04:36:48.506 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:48.507 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:48.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4e1c02ff60 msgr2=0x7f4e1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:48.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 --2- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4e1c02ff60 0x7f4e1c032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f4e180096a0 tx=0x7f4e18006b40).stop 2023-11-15T04:36:48.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 msgr2=0x7f4e30105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:48.508 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 --2- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 0x7f4e30105630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f4e20013510 tx=0x7f4e2000b040).stop 2023-11-15T04:36:48.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 shutdown_connections 2023-11-15T04:36:48.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 --2- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f4e1c02ff60 0x7f4e1c032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:48.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 --2- 172.21.15.190:0/416142028 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e300f5410 0x7f4e30105630 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:48.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 >> 172.21.15.190:0/416142028 conn(0x7f4e300f0b70 msgr2=0x7f4e300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:48.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 shutdown_connections 2023-11-15T04:36:48.510 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:48.507+0000 7f4e34cfc700 1 -- 172.21.15.190:0/416142028 wait complete. 2023-11-15T04:36:48.510 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:49.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:48 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:48.506757+0000 mon.smithi047 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.190:0/416142028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:49.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:49.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:50.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:49 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:49.131102+0000 mgr.smithi047.zgysuz (mgr.14164) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:51.073 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.072+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2170923140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 msgr2=0x7f66f80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:51.074 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.072+0000 7f66fd6a2700 1 --2- 172.21.15.190:0/2170923140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 0x7f66f80f4be0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f66e0009fb0 tx=0x7f66e0009b30).stop 2023-11-15T04:36:51.074 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.072+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2170923140 shutdown_connections 2023-11-15T04:36:51.074 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.072+0000 7f66fd6a2700 1 --2- 172.21.15.190:0/2170923140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 0x7f66f80f4be0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:51.074 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.072+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2170923140 >> 172.21.15.190:0/2170923140 conn(0x7f66f80f02e0 msgr2=0x7f66f80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:51.074 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.072+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2170923140 shutdown_connections 2023-11-15T04:36:51.075 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.072+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2170923140 wait complete. 2023-11-15T04:36:51.076 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.073+0000 7f66fd6a2700 1 Processor -- start 2023-11-15T04:36:51.077 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.073+0000 7f66fd6a2700 1 -- start start 2023-11-15T04:36:51.077 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.073+0000 7f66fd6a2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 0x7f66f80f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:51.077 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.073+0000 7f66fd6a2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f80f57f0 con 0x7f66f80f47c0 2023-11-15T04:36:51.077 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.073+0000 7f66f6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 0x7f66f80f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:51.078 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.073+0000 7f66f6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 0x7f66f80f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56454/0 (socket says 172.21.15.190:56454) 2023-11-15T04:36:51.078 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.073+0000 7f66f6ffd700 1 -- 172.21.15.190:0/2944449140 learned_addr learned my addr 172.21.15.190:0/2944449140 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:51.078 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.074+0000 7f66f6ffd700 1 -- 172.21.15.190:0/2944449140 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e0005040 con 0x7f66f80f47c0 2023-11-15T04:36:51.078 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.074+0000 7f66f6ffd700 1 --2- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 0x7f66f80f7370 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f66e0013510 tx=0x7f66e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:51.078 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.074+0000 7f66ef7fe700 1 -- 172.21.15.190:0/2944449140 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66e0009320 con 0x7f66f80f47c0 2023-11-15T04:36:51.079 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.074+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f80f5990 con 0x7f66f80f47c0 2023-11-15T04:36:51.079 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.074+0000 7f66ef7fe700 1 -- 172.21.15.190:0/2944449140 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66e0013f00 con 0x7f66f80f47c0 2023-11-15T04:36:51.079 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.074+0000 7f66ef7fe700 1 -- 172.21.15.190:0/2944449140 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66e0014d90 con 0x7f66f80f47c0 2023-11-15T04:36:51.079 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.074+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f80f5e20 con 0x7f66f80f47c0 2023-11-15T04:36:51.080 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.075+0000 7f66ef7fe700 1 -- 172.21.15.190:0/2944449140 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66e00095f0 con 0x7f66f80f47c0 2023-11-15T04:36:51.080 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.075+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f8059100 con 0x7f66f80f47c0 2023-11-15T04:36:51.080 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.075+0000 7f66ef7fe700 1 --2- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f66e402ff60 0x7f66e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:51.080 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.075+0000 7f66ef7fe700 1 -- 172.21.15.190:0/2944449140 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66e0041000 con 0x7f66f80f47c0 2023-11-15T04:36:51.080 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.075+0000 7f66f67fc700 1 --2- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f66e402ff60 0x7f66e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:51.081 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.076+0000 7f66f67fc700 1 --2- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f66e402ff60 0x7f66e4032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f66e80096a0 tx=0x7f66e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:51.081 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.080+0000 7f66ef7fe700 1 -- 172.21.15.190:0/2944449140 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66e0014420 con 0x7f66f80f47c0 2023-11-15T04:36:51.285 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.284+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66f80fbf60 con 0x7f66f80f47c0 2023-11-15T04:36:51.285 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.284+0000 7f66ef7fe700 1 -- 172.21.15.190:0/2944449140 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66e0010230 con 0x7f66f80f47c0 2023-11-15T04:36:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:51.286 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:51.287 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f66e402ff60 msgr2=0x7f66e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:51.288 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 --2- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f66e402ff60 0x7f66e4032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f66e80096a0 tx=0x7f66e8006b40).stop 2023-11-15T04:36:51.288 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 msgr2=0x7f66f80f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:51.288 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 --2- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 0x7f66f80f7370 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f66e0013510 tx=0x7f66e000b040).stop 2023-11-15T04:36:51.288 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 shutdown_connections 2023-11-15T04:36:51.288 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 --2- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f66e402ff60 0x7f66e4032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:51.289 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 --2- 172.21.15.190:0/2944449140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66f80f47c0 0x7f66f80f7370 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:51.289 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 >> 172.21.15.190:0/2944449140 conn(0x7f66f80f02e0 msgr2=0x7f66f80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:51.289 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 shutdown_connections 2023-11-15T04:36:51.289 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:51.286+0000 7f66fd6a2700 1 -- 172.21.15.190:0/2944449140 wait complete. 2023-11-15T04:36:51.289 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:51.131502+0000 mgr.smithi047.zgysuz (mgr.14164) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:51.286104+0000 mon.smithi047 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.190:0/2944449140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:52.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:52.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:53.864 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.863+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/4126105883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 msgr2=0x7fca9c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:53.865 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.863+0000 7fcaa1bd5700 1 --2- 172.21.15.190:0/4126105883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 0x7fca9c0f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fca8c009fb0 tx=0x7fca8c009b30).stop 2023-11-15T04:36:53.865 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.863+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/4126105883 shutdown_connections 2023-11-15T04:36:53.865 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.863+0000 7fcaa1bd5700 1 --2- 172.21.15.190:0/4126105883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 0x7fca9c0f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:53.865 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.863+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/4126105883 >> 172.21.15.190:0/4126105883 conn(0x7fca9c0f1c70 msgr2=0x7fca9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:53.865 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.863+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/4126105883 shutdown_connections 2023-11-15T04:36:53.866 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.863+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/4126105883 wait complete. 2023-11-15T04:36:53.867 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.864+0000 7fcaa1bd5700 1 Processor -- start 2023-11-15T04:36:53.868 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.864+0000 7fcaa1bd5700 1 -- start start 2023-11-15T04:36:53.868 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.864+0000 7fcaa1bd5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 0x7fca9c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:53.868 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.864+0000 7fcaa1bd5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c106c20 con 0x7fca9c0f6510 2023-11-15T04:36:53.868 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fca9b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 0x7fca9c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:53.868 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fca9b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 0x7fca9c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56472/0 (socket says 172.21.15.190:56472) 2023-11-15T04:36:53.869 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fca9b7fe700 1 -- 172.21.15.190:0/1768191926 learned_addr learned my addr 172.21.15.190:0/1768191926 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:53.869 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fca9b7fe700 1 -- 172.21.15.190:0/1768191926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c005040 con 0x7fca9c0f6510 2023-11-15T04:36:53.869 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fca9b7fe700 1 --2- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 0x7fca9c106730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fca8c013510 tx=0x7fca8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:53.869 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fca93fff700 1 -- 172.21.15.190:0/1768191926 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca8c009320 con 0x7fca9c0f6510 2023-11-15T04:36:53.869 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca9c106e20 con 0x7fca9c0f6510 2023-11-15T04:36:53.870 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fca93fff700 1 -- 172.21.15.190:0/1768191926 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca8c013f00 con 0x7fca9c0f6510 2023-11-15T04:36:53.870 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fca93fff700 1 -- 172.21.15.190:0/1768191926 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca8c014d90 con 0x7fca9c0f6510 2023-11-15T04:36:53.870 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.865+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca9c1072b0 con 0x7fca9c0f6510 2023-11-15T04:36:53.870 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.866+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9c045ad0 con 0x7fca9c0f6510 2023-11-15T04:36:53.870 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.866+0000 7fca93fff700 1 -- 172.21.15.190:0/1768191926 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca8c0095f0 con 0x7fca9c0f6510 2023-11-15T04:36:53.871 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.866+0000 7fca93fff700 1 --2- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca8802ff60 0x7fca88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:53.871 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.866+0000 7fca93fff700 1 -- 172.21.15.190:0/1768191926 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca8c040fe0 con 0x7fca9c0f6510 2023-11-15T04:36:53.871 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.867+0000 7fca9affd700 1 --2- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca8802ff60 0x7fca88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:53.871 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.867+0000 7fca9affd700 1 --2- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca8802ff60 0x7fca88032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fca840096a0 tx=0x7fca84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:53.872 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:53.871+0000 7fca93fff700 1 -- 172.21.15.190:0/1768191926 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca8c010230 con 0x7fca9c0f6510 2023-11-15T04:36:54.076 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.075+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca9c0fbf60 con 0x7fca9c0f6510 2023-11-15T04:36:54.076 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.076+0000 7fca93fff700 1 -- 172.21.15.190:0/1768191926 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca8c010230 con 0x7fca9c0f6510 2023-11-15T04:36:54.077 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:54.077 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:54.078 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca8802ff60 msgr2=0x7fca88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:54.078 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 --2- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca8802ff60 0x7fca88032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fca840096a0 tx=0x7fca84006b40).stop 2023-11-15T04:36:54.078 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 msgr2=0x7fca9c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:54.079 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 --2- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 0x7fca9c106730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fca8c013510 tx=0x7fca8c00b040).stop 2023-11-15T04:36:54.079 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 shutdown_connections 2023-11-15T04:36:54.079 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 --2- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fca8802ff60 0x7fca88032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:54.079 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 --2- 172.21.15.190:0/1768191926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca9c0f6510 0x7fca9c106730 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:54.079 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 >> 172.21.15.190:0/1768191926 conn(0x7fca9c0f1c70 msgr2=0x7fca9c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:54.080 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 shutdown_connections 2023-11-15T04:36:54.080 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:54.077+0000 7fcaa1bd5700 1 -- 172.21.15.190:0/1768191926 wait complete. 2023-11-15T04:36:54.080 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:53.131866+0000 mgr.smithi047.zgysuz (mgr.14164) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:54.077180+0000 mon.smithi047 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.190:0/1768191926' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:55.480 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:55.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:55.132257+0000 mgr.smithi047.zgysuz (mgr.14164) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:56.631 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 -- 172.21.15.190:0/2906599198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 msgr2=0x7f3b2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:56.631 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 --2- 172.21.15.190:0/2906599198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 0x7f3b2c0f6930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f3b1c009fb0 tx=0x7f3b1c009b30).stop 2023-11-15T04:36:56.631 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 -- 172.21.15.190:0/2906599198 shutdown_connections 2023-11-15T04:36:56.631 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 --2- 172.21.15.190:0/2906599198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 0x7f3b2c0f6930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:56.632 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 -- 172.21.15.190:0/2906599198 >> 172.21.15.190:0/2906599198 conn(0x7f3b2c0f1c70 msgr2=0x7f3b2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:56.632 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 -- 172.21.15.190:0/2906599198 shutdown_connections 2023-11-15T04:36:56.632 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 -- 172.21.15.190:0/2906599198 wait complete. 2023-11-15T04:36:56.632 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 Processor -- start 2023-11-15T04:36:56.633 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 -- start start 2023-11-15T04:36:56.633 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 0x7f3b2c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:56.633 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.628+0000 7f3b33eb5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c069e20 con 0x7f3b2c0f6510 2023-11-15T04:36:56.633 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.629+0000 7f3b31c51700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 0x7f3b2c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:56.634 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.629+0000 7f3b31c51700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 0x7f3b2c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56480/0 (socket says 172.21.15.190:56480) 2023-11-15T04:36:56.634 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.629+0000 7f3b31c51700 1 -- 172.21.15.190:0/493490054 learned_addr learned my addr 172.21.15.190:0/493490054 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:56.634 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.629+0000 7f3b31c51700 1 -- 172.21.15.190:0/493490054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c005040 con 0x7f3b2c0f6510 2023-11-15T04:36:56.634 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.629+0000 7f3b31c51700 1 --2- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 0x7f3b2c06b9a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3b1c013510 tx=0x7f3b1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:56.634 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.629+0000 7f3b227fc700 1 -- 172.21.15.190:0/493490054 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b1c009320 con 0x7f3b2c0f6510 2023-11-15T04:36:56.635 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.629+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c069fc0 con 0x7f3b2c0f6510 2023-11-15T04:36:56.635 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.629+0000 7f3b227fc700 1 -- 172.21.15.190:0/493490054 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b1c013f00 con 0x7f3b2c0f6510 2023-11-15T04:36:56.635 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.629+0000 7f3b227fc700 1 -- 172.21.15.190:0/493490054 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b1c014d90 con 0x7f3b2c0f6510 2023-11-15T04:36:56.635 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.630+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c06a450 con 0x7f3b2c0f6510 2023-11-15T04:36:56.636 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.630+0000 7f3b227fc700 1 -- 172.21.15.190:0/493490054 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b1c0095f0 con 0x7f3b2c0f6510 2023-11-15T04:36:56.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.630+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c059100 con 0x7f3b2c0f6510 2023-11-15T04:36:56.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.630+0000 7f3b227fc700 1 --2- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3b1802ff60 0x7f3b18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:56.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.631+0000 7f3b227fc700 1 -- 172.21.15.190:0/493490054 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b1c0410c0 con 0x7f3b2c0f6510 2023-11-15T04:36:56.637 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.631+0000 7f3b31450700 1 --2- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3b1802ff60 0x7f3b18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:56.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.631+0000 7f3b31450700 1 --2- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3b1802ff60 0x7f3b18032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f3b280096a0 tx=0x7f3b28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:56.638 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.636+0000 7f3b227fc700 1 -- 172.21.15.190:0/493490054 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b1c014420 con 0x7f3b2c0f6510 2023-11-15T04:36:56.843 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.841+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b2c0fbf60 con 0x7f3b2c0f6510 2023-11-15T04:36:56.843 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.842+0000 7f3b227fc700 1 -- 172.21.15.190:0/493490054 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b1c010230 con 0x7f3b2c0f6510 2023-11-15T04:36:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:56.844 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:56.845 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3b1802ff60 msgr2=0x7f3b18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:56.845 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 --2- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3b1802ff60 0x7f3b18032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f3b280096a0 tx=0x7f3b28006b40).stop 2023-11-15T04:36:56.845 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 msgr2=0x7f3b2c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:56.845 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 --2- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 0x7f3b2c06b9a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3b1c013510 tx=0x7f3b1c00b040).stop 2023-11-15T04:36:56.845 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 shutdown_connections 2023-11-15T04:36:56.846 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 --2- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f3b1802ff60 0x7f3b18032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:56.846 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 --2- 172.21.15.190:0/493490054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b2c0f6510 0x7f3b2c06b9a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:56.846 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 >> 172.21.15.190:0/493490054 conn(0x7f3b2c0f1c70 msgr2=0x7f3b2c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:56.846 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 shutdown_connections 2023-11-15T04:36:56.847 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:56.844+0000 7f3b33eb5700 1 -- 172.21.15.190:0/493490054 wait complete. 2023-11-15T04:36:56.847 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:36:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:56.843729+0000 mon.smithi047 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.190:0/493490054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:36:58.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:36:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:57.132618+0000 mgr.smithi047.zgysuz (mgr.14164) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:36:58.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:36:58.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:36:59.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.394+0000 7f29ba0f9700 1 -- 172.21.15.190:0/105525107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 msgr2=0x7f29b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:59.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.394+0000 7f29ba0f9700 1 --2- 172.21.15.190:0/105525107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 0x7f29b40f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f299c009fb0 tx=0x7f299c009b30).stop 2023-11-15T04:36:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.394+0000 7f29ba0f9700 1 -- 172.21.15.190:0/105525107 shutdown_connections 2023-11-15T04:36:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.394+0000 7f29ba0f9700 1 --2- 172.21.15.190:0/105525107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 0x7f29b40f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.394+0000 7f29ba0f9700 1 -- 172.21.15.190:0/105525107 >> 172.21.15.190:0/105525107 conn(0x7f29b40f0b70 msgr2=0x7f29b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.394+0000 7f29ba0f9700 1 -- 172.21.15.190:0/105525107 shutdown_connections 2023-11-15T04:36:59.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.394+0000 7f29ba0f9700 1 -- 172.21.15.190:0/105525107 wait complete. 2023-11-15T04:36:59.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.395+0000 7f29ba0f9700 1 Processor -- start 2023-11-15T04:36:59.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.395+0000 7f29ba0f9700 1 -- start start 2023-11-15T04:36:59.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.395+0000 7f29ba0f9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 0x7f29b4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:59.398 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.395+0000 7f29ba0f9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b4105b20 con 0x7f29b40f5410 2023-11-15T04:36:59.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29b37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 0x7f29b4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:59.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29b37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 0x7f29b4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:56500/0 (socket says 172.21.15.190:56500) 2023-11-15T04:36:59.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29b37fe700 1 -- 172.21.15.190:0/2020284814 learned_addr learned my addr 172.21.15.190:0/2020284814 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:36:59.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29b37fe700 1 -- 172.21.15.190:0/2020284814 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c005040 con 0x7f29b40f5410 2023-11-15T04:36:59.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29b37fe700 1 --2- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 0x7f29b4105630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f299c013510 tx=0x7f299c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:59.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29abfff700 1 -- 172.21.15.190:0/2020284814 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f299c009320 con 0x7f29b40f5410 2023-11-15T04:36:59.401 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b4105d20 con 0x7f29b40f5410 2023-11-15T04:36:59.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29abfff700 1 -- 172.21.15.190:0/2020284814 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f299c013f00 con 0x7f29b40f5410 2023-11-15T04:36:59.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29abfff700 1 -- 172.21.15.190:0/2020284814 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f299c014d90 con 0x7f29b40f5410 2023-11-15T04:36:59.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.396+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b41061b0 con 0x7f29b40f5410 2023-11-15T04:36:59.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.397+0000 7f29abfff700 1 -- 172.21.15.190:0/2020284814 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f299c0095f0 con 0x7f29b40f5410 2023-11-15T04:36:59.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.397+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b4045ad0 con 0x7f29b40f5410 2023-11-15T04:36:59.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.397+0000 7f29abfff700 1 --2- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f29a002ff60 0x7f29a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:36:59.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.397+0000 7f29abfff700 1 -- 172.21.15.190:0/2020284814 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f299c040fd0 con 0x7f29b40f5410 2023-11-15T04:36:59.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.398+0000 7f29b2ffd700 1 --2- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f29a002ff60 0x7f29a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:36:59.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.398+0000 7f29b2ffd700 1 --2- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f29a002ff60 0x7f29a0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f29a40096a0 tx=0x7f29a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:36:59.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.402+0000 7f29abfff700 1 -- 172.21.15.190:0/2020284814 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f299c014420 con 0x7f29b40f5410 2023-11-15T04:36:59.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.608+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29b40fae60 con 0x7f29b40f5410 2023-11-15T04:36:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.609+0000 7f29abfff700 1 -- 172.21.15.190:0/2020284814 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f299c01d7f0 con 0x7f29b40f5410 2023-11-15T04:36:59.610 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:36:59.610 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:36:59.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.610+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f29a002ff60 msgr2=0x7f29a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.610+0000 7f29ba0f9700 1 --2- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f29a002ff60 0x7f29a0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f29a40096a0 tx=0x7f29a4006b40).stop 2023-11-15T04:36:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.610+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 msgr2=0x7f29b4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:36:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.610+0000 7f29ba0f9700 1 --2- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 0x7f29b4105630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f299c013510 tx=0x7f299c00b040).stop 2023-11-15T04:36:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.610+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 shutdown_connections 2023-11-15T04:36:59.613 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.610+0000 7f29ba0f9700 1 --2- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f29a002ff60 0x7f29a0032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:59.613 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.610+0000 7f29ba0f9700 1 --2- 172.21.15.190:0/2020284814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b40f5410 0x7f29b4105630 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:36:59.613 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.610+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 >> 172.21.15.190:0/2020284814 conn(0x7f29b40f0b70 msgr2=0x7f29b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:36:59.613 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.611+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 shutdown_connections 2023-11-15T04:36:59.613 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:36:59.611+0000 7f29ba0f9700 1 -- 172.21.15.190:0/2020284814 wait complete. 2023-11-15T04:36:59.614 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:36:59.132873+0000 mgr.smithi047.zgysuz (mgr.14164) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:36:59.610387+0000 mon.smithi047 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.190:0/2020284814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:00.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:00.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:02.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.185+0000 7fb640a21700 1 -- 172.21.15.190:0/2901241047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 msgr2=0x7fb63c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:02.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.185+0000 7fb640a21700 1 --2- 172.21.15.190:0/2901241047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 0x7fb63c0f90f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fb630009fb0 tx=0x7fb630009b30).stop 2023-11-15T04:37:02.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.185+0000 7fb640a21700 1 -- 172.21.15.190:0/2901241047 shutdown_connections 2023-11-15T04:37:02.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.185+0000 7fb640a21700 1 --2- 172.21.15.190:0/2901241047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 0x7fb63c0f90f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:02.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.185+0000 7fb640a21700 1 -- 172.21.15.190:0/2901241047 >> 172.21.15.190:0/2901241047 conn(0x7fb63c0f0b70 msgr2=0x7fb63c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:02.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.185+0000 7fb640a21700 1 -- 172.21.15.190:0/2901241047 shutdown_connections 2023-11-15T04:37:02.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.185+0000 7fb640a21700 1 -- 172.21.15.190:0/2901241047 wait complete. 2023-11-15T04:37:02.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.185+0000 7fb640a21700 1 Processor -- start 2023-11-15T04:37:02.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.185+0000 7fb640a21700 1 -- start start 2023-11-15T04:37:02.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.186+0000 7fb640a21700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 0x7fb63c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:02.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.186+0000 7fb640a21700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c106bd0 con 0x7fb63c060280 2023-11-15T04:37:02.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.186+0000 7fb63a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 0x7fb63c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:02.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.186+0000 7fb63a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 0x7fb63c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:33784/0 (socket says 172.21.15.190:33784) 2023-11-15T04:37:02.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.186+0000 7fb63a59c700 1 -- 172.21.15.190:0/1754654284 learned_addr learned my addr 172.21.15.190:0/1754654284 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:02.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.186+0000 7fb63a59c700 1 -- 172.21.15.190:0/1754654284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb630005040 con 0x7fb63c060280 2023-11-15T04:37:02.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.186+0000 7fb63a59c700 1 --2- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 0x7fb63c1066e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fb630013510 tx=0x7fb63000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:02.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.187+0000 7fb62affd700 1 -- 172.21.15.190:0/1754654284 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb630009320 con 0x7fb63c060280 2023-11-15T04:37:02.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.187+0000 7fb62affd700 1 -- 172.21.15.190:0/1754654284 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb630013f00 con 0x7fb63c060280 2023-11-15T04:37:02.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.187+0000 7fb62affd700 1 -- 172.21.15.190:0/1754654284 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb630014dc0 con 0x7fb63c060280 2023-11-15T04:37:02.192 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.187+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63c106dd0 con 0x7fb63c060280 2023-11-15T04:37:02.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.187+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63c107260 con 0x7fb63c060280 2023-11-15T04:37:02.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.188+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63c045ad0 con 0x7fb63c060280 2023-11-15T04:37:02.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.190+0000 7fb62affd700 1 -- 172.21.15.190:0/1754654284 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6300095f0 con 0x7fb63c060280 2023-11-15T04:37:02.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.190+0000 7fb62affd700 1 --2- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb624030000 0x7fb6240324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:02.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.190+0000 7fb62affd700 1 -- 172.21.15.190:0/1754654284 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb630041230 con 0x7fb63c060280 2023-11-15T04:37:02.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.190+0000 7fb639d9b700 1 --2- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb624030000 0x7fb6240324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:02.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.191+0000 7fb639d9b700 1 --2- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb624030000 0x7fb6240324c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb62c0096a0 tx=0x7fb62c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:02.194 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.194+0000 7fb62affd700 1 -- 172.21.15.190:0/1754654284 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb630014420 con 0x7fb63c060280 2023-11-15T04:37:02.399 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.398+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb63c108110 con 0x7fb63c060280 2023-11-15T04:37:02.400 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.399+0000 7fb62affd700 1 -- 172.21.15.190:0/1754654284 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb63002b030 con 0x7fb63c060280 2023-11-15T04:37:02.400 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:02.401 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:02.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb624030000 msgr2=0x7fb6240324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:02.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 --2- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb624030000 0x7fb6240324c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb62c0096a0 tx=0x7fb62c006b40).stop 2023-11-15T04:37:02.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 msgr2=0x7fb63c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:02.402 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 --2- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 0x7fb63c1066e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fb630013510 tx=0x7fb63000b040).stop 2023-11-15T04:37:02.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 shutdown_connections 2023-11-15T04:37:02.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 --2- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb624030000 0x7fb6240324c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:02.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 --2- 172.21.15.190:0/1754654284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb63c060280 0x7fb63c1066e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:02.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 >> 172.21.15.190:0/1754654284 conn(0x7fb63c0f0b70 msgr2=0x7fb63c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:02.403 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 shutdown_connections 2023-11-15T04:37:02.404 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:02.401+0000 7fb640a21700 1 -- 172.21.15.190:0/1754654284 wait complete. 2023-11-15T04:37:02.404 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:02.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:02 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:01.133096+0000 mgr.smithi047.zgysuz (mgr.14164) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:03.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:03 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:02.400774+0000 mon.smithi047 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.190:0/1754654284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:03.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:03.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:04.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:04 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:03.133339+0000 mgr.smithi047.zgysuz (mgr.14164) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:04.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.899+0000 7f1f32cd7700 1 -- 172.21.15.190:0/3496642967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 msgr2=0x7f1f2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:04.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.899+0000 7f1f32cd7700 1 --2- 172.21.15.190:0/3496642967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 0x7f1f2c0f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f1f20009fb0 tx=0x7f1f20009b30).stop 2023-11-15T04:37:04.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.899+0000 7f1f32cd7700 1 -- 172.21.15.190:0/3496642967 shutdown_connections 2023-11-15T04:37:04.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.899+0000 7f1f32cd7700 1 --2- 172.21.15.190:0/3496642967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 0x7f1f2c0f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:04.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.899+0000 7f1f32cd7700 1 -- 172.21.15.190:0/3496642967 >> 172.21.15.190:0/3496642967 conn(0x7f1f2c0f0b70 msgr2=0x7f1f2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:04.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.899+0000 7f1f32cd7700 1 -- 172.21.15.190:0/3496642967 shutdown_connections 2023-11-15T04:37:04.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.899+0000 7f1f32cd7700 1 -- 172.21.15.190:0/3496642967 wait complete. 2023-11-15T04:37:04.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.899+0000 7f1f32cd7700 1 Processor -- start 2023-11-15T04:37:04.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.900+0000 7f1f32cd7700 1 -- start start 2023-11-15T04:37:04.903 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.900+0000 7f1f32cd7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 0x7f1f2c1093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:04.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.900+0000 7f1f32cd7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c060280 con 0x7f1f2c0f5410 2023-11-15T04:37:04.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.900+0000 7f1f30a73700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 0x7f1f2c1093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:04.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.900+0000 7f1f30a73700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 0x7f1f2c1093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:33802/0 (socket says 172.21.15.190:33802) 2023-11-15T04:37:04.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.900+0000 7f1f30a73700 1 -- 172.21.15.190:0/1366206247 learned_addr learned my addr 172.21.15.190:0/1366206247 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:04.904 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.900+0000 7f1f30a73700 1 -- 172.21.15.190:0/1366206247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20005040 con 0x7f1f2c0f5410 2023-11-15T04:37:04.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.900+0000 7f1f30a73700 1 --2- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 0x7f1f2c1093c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f1f20013510 tx=0x7f1f2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:04.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.901+0000 7f1f297fa700 1 -- 172.21.15.190:0/1366206247 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f20009320 con 0x7f1f2c0f5410 2023-11-15T04:37:04.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.901+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c060480 con 0x7f1f2c0f5410 2023-11-15T04:37:04.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.901+0000 7f1f297fa700 1 -- 172.21.15.190:0/1366206247 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f20013f00 con 0x7f1f2c0f5410 2023-11-15T04:37:04.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.901+0000 7f1f297fa700 1 -- 172.21.15.190:0/1366206247 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f20014d90 con 0x7f1f2c0f5410 2023-11-15T04:37:04.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.901+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c109af0 con 0x7f1f2c0f5410 2023-11-15T04:37:04.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.901+0000 7f1f297fa700 1 -- 172.21.15.190:0/1366206247 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f200095f0 con 0x7f1f2c0f5410 2023-11-15T04:37:04.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.901+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c045ad0 con 0x7f1f2c0f5410 2023-11-15T04:37:04.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.902+0000 7f1f297fa700 1 --2- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1f1c02ff60 0x7f1f1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:04.908 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.902+0000 7f1f297fa700 1 -- 172.21.15.190:0/1366206247 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f20041000 con 0x7f1f2c0f5410 2023-11-15T04:37:04.908 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.902+0000 7f1f2bfff700 1 --2- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1f1c02ff60 0x7f1f1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:04.908 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.902+0000 7f1f2bfff700 1 --2- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1f1c02ff60 0x7f1f1c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f1f180096a0 tx=0x7f1f18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:04.909 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:04.907+0000 7f1f297fa700 1 -- 172.21.15.190:0/1366206247 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f2001d7f0 con 0x7f1f2c0f5410 2023-11-15T04:37:05.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.110+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f2c10a310 con 0x7f1f2c0f5410 2023-11-15T04:37:05.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.110+0000 7f1f297fa700 1 -- 172.21.15.190:0/1366206247 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f20010230 con 0x7f1f2c0f5410 2023-11-15T04:37:05.112 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:05.112 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:05.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1f1c02ff60 msgr2=0x7f1f1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:05.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 --2- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1f1c02ff60 0x7f1f1c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f1f180096a0 tx=0x7f1f18006b40).stop 2023-11-15T04:37:05.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 msgr2=0x7f1f2c1093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:05.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 --2- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 0x7f1f2c1093c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f1f20013510 tx=0x7f1f2000b040).stop 2023-11-15T04:37:05.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 shutdown_connections 2023-11-15T04:37:05.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 --2- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f1f1c02ff60 0x7f1f1c032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:05.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 --2- 172.21.15.190:0/1366206247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f2c0f5410 0x7f1f2c1093c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:05.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 >> 172.21.15.190:0/1366206247 conn(0x7f1f2c0f0b70 msgr2=0x7f1f2c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:05.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 shutdown_connections 2023-11-15T04:37:05.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:05.112+0000 7f1f32cd7700 1 -- 172.21.15.190:0/1366206247 wait complete. 2023-11-15T04:37:05.116 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:05.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:05 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:05.112056+0000 mon.smithi047 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.190:0/1366206247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:06.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:06.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:05.133705+0000 mgr.smithi047.zgysuz (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:05.222069+0000 mon.smithi047 (mon.0) 305 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:37:06.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:06 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:05.222937+0000 mon.smithi047 (mon.0) 306 : audit [INF] from='mgr.14164 172.21.15.47:0/331580570' entity='mgr.smithi047.zgysuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.zgysuz/trash_purge_schedule"}]: dispatch 2023-11-15T04:37:07.605 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.603+0000 7efc29f5c700 1 -- 172.21.15.190:0/168678536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 msgr2=0x7efc240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:07.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.603+0000 7efc29f5c700 1 --2- 172.21.15.190:0/168678536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 0x7efc240f90f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7efc18009fb0 tx=0x7efc18009b30).stop 2023-11-15T04:37:07.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.603+0000 7efc29f5c700 1 -- 172.21.15.190:0/168678536 shutdown_connections 2023-11-15T04:37:07.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.603+0000 7efc29f5c700 1 --2- 172.21.15.190:0/168678536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 0x7efc240f90f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:07.606 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.603+0000 7efc29f5c700 1 -- 172.21.15.190:0/168678536 >> 172.21.15.190:0/168678536 conn(0x7efc240f0b70 msgr2=0x7efc240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:07.607 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.603+0000 7efc29f5c700 1 -- 172.21.15.190:0/168678536 shutdown_connections 2023-11-15T04:37:07.607 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.603+0000 7efc29f5c700 1 -- 172.21.15.190:0/168678536 wait complete. 2023-11-15T04:37:07.607 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.604+0000 7efc29f5c700 1 Processor -- start 2023-11-15T04:37:07.607 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.604+0000 7efc29f5c700 1 -- start start 2023-11-15T04:37:07.607 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.604+0000 7efc29f5c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 0x7efc241066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:07.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.604+0000 7efc29f5c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc24106bd0 con 0x7efc24060280 2023-11-15T04:37:07.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.604+0000 7efc237fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 0x7efc241066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:07.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.604+0000 7efc237fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 0x7efc241066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:33816/0 (socket says 172.21.15.190:33816) 2023-11-15T04:37:07.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.604+0000 7efc237fe700 1 -- 172.21.15.190:0/3739238500 learned_addr learned my addr 172.21.15.190:0/3739238500 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:07.608 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.605+0000 7efc237fe700 1 -- 172.21.15.190:0/3739238500 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc18005040 con 0x7efc24060280 2023-11-15T04:37:07.609 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.605+0000 7efc237fe700 1 --2- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 0x7efc241066e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7efc18013510 tx=0x7efc1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:07.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.605+0000 7efc0bfff700 1 -- 172.21.15.190:0/3739238500 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc18009320 con 0x7efc24060280 2023-11-15T04:37:07.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.605+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc24106dd0 con 0x7efc24060280 2023-11-15T04:37:07.610 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.605+0000 7efc0bfff700 1 -- 172.21.15.190:0/3739238500 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc18013f00 con 0x7efc24060280 2023-11-15T04:37:07.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.605+0000 7efc0bfff700 1 -- 172.21.15.190:0/3739238500 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc18014dc0 con 0x7efc24060280 2023-11-15T04:37:07.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.605+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc24107260 con 0x7efc24060280 2023-11-15T04:37:07.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.606+0000 7efc0bfff700 1 -- 172.21.15.190:0/3739238500 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc1801d3d0 con 0x7efc24060280 2023-11-15T04:37:07.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.606+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc24045ad0 con 0x7efc24060280 2023-11-15T04:37:07.611 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.606+0000 7efc0bfff700 1 --2- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efc0c02ff60 0x7efc0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:07.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.606+0000 7efc0bfff700 1 -- 172.21.15.190:0/3739238500 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc180412f0 con 0x7efc24060280 2023-11-15T04:37:07.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.606+0000 7efc22ffd700 1 --2- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efc0c02ff60 0x7efc0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:07.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.607+0000 7efc22ffd700 1 --2- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efc0c02ff60 0x7efc0c032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7efc140096a0 tx=0x7efc14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:07.612 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.611+0000 7efc0bfff700 1 -- 172.21.15.190:0/3739238500 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc1801ccb0 con 0x7efc24060280 2023-11-15T04:37:07.819 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.818+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc24108110 con 0x7efc24060280 2023-11-15T04:37:07.820 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.819+0000 7efc0bfff700 1 -- 172.21.15.190:0/3739238500 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc18012330 con 0x7efc24060280 2023-11-15T04:37:07.820 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:07.821 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:07.821 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efc0c02ff60 msgr2=0x7efc0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:07.822 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 --2- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efc0c02ff60 0x7efc0c032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7efc140096a0 tx=0x7efc14006b40).stop 2023-11-15T04:37:07.822 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 msgr2=0x7efc241066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:07.822 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 --2- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 0x7efc241066e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7efc18013510 tx=0x7efc1800b040).stop 2023-11-15T04:37:07.822 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 shutdown_connections 2023-11-15T04:37:07.823 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 --2- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7efc0c02ff60 0x7efc0c032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:07.823 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 --2- 172.21.15.190:0/3739238500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24060280 0x7efc241066e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:07.823 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 >> 172.21.15.190:0/3739238500 conn(0x7efc240f0b70 msgr2=0x7efc240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:07.823 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 shutdown_connections 2023-11-15T04:37:07.823 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:07.821+0000 7efc29f5c700 1 -- 172.21.15.190:0/3739238500 wait complete. 2023-11-15T04:37:07.824 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:08.007 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:07 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:07.134048+0000 mgr.smithi047.zgysuz (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:09.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:09.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:09.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:08 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:07.820514+0000 mon.smithi047 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.190:0/3739238500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:10.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:09 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:09.134392+0000 mgr.smithi047.zgysuz (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:10.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.264+0000 7f7ae159a700 1 -- 172.21.15.190:0/1293695834 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7acc0138a0 con 0x7f7ae40f62a0 2023-11-15T04:37:10.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.265+0000 7f7ae359e700 1 -- 172.21.15.190:0/1293695834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 msgr2=0x7f7ae40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:10.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.265+0000 7f7ae359e700 1 --2- 172.21.15.190:0/1293695834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 0x7f7ae40f66c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f7acc009fb0 tx=0x7f7acc009b30).stop 2023-11-15T04:37:10.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.265+0000 7f7ae359e700 1 -- 172.21.15.190:0/1293695834 shutdown_connections 2023-11-15T04:37:10.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.265+0000 7f7ae359e700 1 --2- 172.21.15.190:0/1293695834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 0x7f7ae40f66c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:10.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.265+0000 7f7ae359e700 1 -- 172.21.15.190:0/1293695834 >> 172.21.15.190:0/1293695834 conn(0x7f7ae40f1ac0 msgr2=0x7f7ae40f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:10.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.265+0000 7f7ae359e700 1 -- 172.21.15.190:0/1293695834 shutdown_connections 2023-11-15T04:37:10.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.265+0000 7f7ae359e700 1 -- 172.21.15.190:0/1293695834 wait complete. 2023-11-15T04:37:10.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.265+0000 7f7ae359e700 1 Processor -- start 2023-11-15T04:37:10.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.265+0000 7f7ae359e700 1 -- start start 2023-11-15T04:37:10.269 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7ae359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 0x7f7ae41064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:10.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7ae359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae4106990 con 0x7f7ae40f62a0 2023-11-15T04:37:10.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7ae259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 0x7f7ae41064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:10.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7ae259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 0x7f7ae41064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:52912/0 (socket says 172.21.15.190:52912) 2023-11-15T04:37:10.270 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7ae259c700 1 -- 172.21.15.190:0/2304402305 learned_addr learned my addr 172.21.15.190:0/2304402305 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:10.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7ae259c700 1 -- 172.21.15.190:0/2304402305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc005040 con 0x7f7ae40f62a0 2023-11-15T04:37:10.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7ae259c700 1 --2- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 0x7f7ae41064a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f7acc013510 tx=0x7f7acc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:10.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7adaffd700 1 -- 172.21.15.190:0/2304402305 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7acc009320 con 0x7f7ae40f62a0 2023-11-15T04:37:10.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae4106b90 con 0x7f7ae40f62a0 2023-11-15T04:37:10.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7adaffd700 1 -- 172.21.15.190:0/2304402305 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7acc013f00 con 0x7f7ae40f62a0 2023-11-15T04:37:10.271 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7adaffd700 1 -- 172.21.15.190:0/2304402305 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7acc014dc0 con 0x7f7ae40f62a0 2023-11-15T04:37:10.272 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.266+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae4107020 con 0x7f7ae40f62a0 2023-11-15T04:37:10.272 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.267+0000 7f7adaffd700 1 -- 172.21.15.190:0/2304402305 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7acc0095f0 con 0x7f7ae40f62a0 2023-11-15T04:37:10.272 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.267+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae4045ad0 con 0x7f7ae40f62a0 2023-11-15T04:37:10.272 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.267+0000 7f7adaffd700 1 --2- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ad002ff60 0x7f7ad0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:10.272 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.267+0000 7f7adaffd700 1 -- 172.21.15.190:0/2304402305 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7acc041000 con 0x7f7ae40f62a0 2023-11-15T04:37:10.273 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.268+0000 7f7ae1d9b700 1 --2- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ad002ff60 0x7f7ad0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:10.273 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.268+0000 7f7ae1d9b700 1 --2- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ad002ff60 0x7f7ad0032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f7ad40096a0 tx=0x7f7ad4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:10.273 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.272+0000 7f7adaffd700 1 -- 172.21.15.190:0/2304402305 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7acc01c790 con 0x7f7ae40f62a0 2023-11-15T04:37:10.478 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.478+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ae4107ed0 con 0x7f7ae40f62a0 2023-11-15T04:37:10.479 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.478+0000 7f7adaffd700 1 -- 172.21.15.190:0/2304402305 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7acc010230 con 0x7f7ae40f62a0 2023-11-15T04:37:10.480 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:10.480 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:10.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ad002ff60 msgr2=0x7f7ad0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:10.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 --2- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ad002ff60 0x7f7ad0032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f7ad40096a0 tx=0x7f7ad4006b40).stop 2023-11-15T04:37:10.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 msgr2=0x7f7ae41064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:10.481 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 --2- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 0x7f7ae41064a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f7acc013510 tx=0x7f7acc00b040).stop 2023-11-15T04:37:10.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 shutdown_connections 2023-11-15T04:37:10.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 --2- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7ad002ff60 0x7f7ad0032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:10.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 --2- 172.21.15.190:0/2304402305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae40f62a0 0x7f7ae41064a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:10.482 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 >> 172.21.15.190:0/2304402305 conn(0x7f7ae40f1ac0 msgr2=0x7f7ae40f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:10.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 shutdown_connections 2023-11-15T04:37:10.483 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:10.480+0000 7f7ae359e700 1 -- 172.21.15.190:0/2304402305 wait complete. 2023-11-15T04:37:10.483 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:11.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:10 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:10.479861+0000 mon.smithi047 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.190:0/2304402305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:11.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:11.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:12.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:11 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:11.134813+0000 mgr.smithi047.zgysuz (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:13.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 -- 172.21.15.190:0/71100695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 msgr2=0x7f7b7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:13.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 --2- 172.21.15.190:0/71100695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 0x7f7b7c0f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f7b6c009fb0 tx=0x7f7b6c009b30).stop 2023-11-15T04:37:13.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 -- 172.21.15.190:0/71100695 shutdown_connections 2023-11-15T04:37:13.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 --2- 172.21.15.190:0/71100695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 0x7f7b7c0f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:13.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 -- 172.21.15.190:0/71100695 >> 172.21.15.190:0/71100695 conn(0x7f7b7c0f0b70 msgr2=0x7f7b7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:13.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 -- 172.21.15.190:0/71100695 shutdown_connections 2023-11-15T04:37:13.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 -- 172.21.15.190:0/71100695 wait complete. 2023-11-15T04:37:13.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 Processor -- start 2023-11-15T04:37:13.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 -- start start 2023-11-15T04:37:13.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 0x7f7b7c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:13.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.015+0000 7f7b83b95700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c060220 con 0x7f7b7c0f5410 2023-11-15T04:37:13.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b81931700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 0x7f7b7c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:13.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b81931700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 0x7f7b7c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:52930/0 (socket says 172.21.15.190:52930) 2023-11-15T04:37:13.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b81931700 1 -- 172.21.15.190:0/1617737847 learned_addr learned my addr 172.21.15.190:0/1617737847 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:13.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b81931700 1 -- 172.21.15.190:0/1617737847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c005040 con 0x7f7b7c0f5410 2023-11-15T04:37:13.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b81931700 1 --2- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 0x7f7b7c109450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f7b6c013510 tx=0x7f7b6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:13.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b727fc700 1 -- 172.21.15.190:0/1617737847 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b6c009320 con 0x7f7b7c0f5410 2023-11-15T04:37:13.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c060420 con 0x7f7b7c0f5410 2023-11-15T04:37:13.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b727fc700 1 -- 172.21.15.190:0/1617737847 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b6c013f00 con 0x7f7b7c0f5410 2023-11-15T04:37:13.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c109b00 con 0x7f7b7c0f5410 2023-11-15T04:37:13.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.016+0000 7f7b727fc700 1 -- 172.21.15.190:0/1617737847 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b6c014db0 con 0x7f7b7c0f5410 2023-11-15T04:37:13.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.017+0000 7f7b727fc700 1 -- 172.21.15.190:0/1617737847 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b6c0095f0 con 0x7f7b7c0f5410 2023-11-15T04:37:13.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.017+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c059100 con 0x7f7b7c0f5410 2023-11-15T04:37:13.023 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.017+0000 7f7b727fc700 1 --2- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b6802ff60 0x7f7b68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:13.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.017+0000 7f7b727fc700 1 -- 172.21.15.190:0/1617737847 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b6c040fe0 con 0x7f7b7c0f5410 2023-11-15T04:37:13.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.018+0000 7f7b81130700 1 --2- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b6802ff60 0x7f7b68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:13.026 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.018+0000 7f7b81130700 1 --2- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b6802ff60 0x7f7b68032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f7b780096a0 tx=0x7f7b78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:13.026 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.023+0000 7f7b727fc700 1 -- 172.21.15.190:0/1617737847 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b6c014420 con 0x7f7b7c0f5410 2023-11-15T04:37:13.232 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.231+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b7c045ad0 con 0x7f7b7c0f5410 2023-11-15T04:37:13.233 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.232+0000 7f7b727fc700 1 -- 172.21.15.190:0/1617737847 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b6c01c920 con 0x7f7b7c0f5410 2023-11-15T04:37:13.233 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:13.233 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:13.234 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b6802ff60 msgr2=0x7f7b68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:13.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 --2- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b6802ff60 0x7f7b68032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f7b780096a0 tx=0x7f7b78006b40).stop 2023-11-15T04:37:13.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 msgr2=0x7f7b7c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:13.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 --2- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 0x7f7b7c109450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f7b6c013510 tx=0x7f7b6c00b040).stop 2023-11-15T04:37:13.235 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 shutdown_connections 2023-11-15T04:37:13.236 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 --2- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f7b6802ff60 0x7f7b68032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:13.236 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 --2- 172.21.15.190:0/1617737847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0f5410 0x7f7b7c109450 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:13.236 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 >> 172.21.15.190:0/1617737847 conn(0x7f7b7c0f0b70 msgr2=0x7f7b7c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:13.236 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 shutdown_connections 2023-11-15T04:37:13.236 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:13.233+0000 7f7b83b95700 1 -- 172.21.15.190:0/1617737847 wait complete. 2023-11-15T04:37:13.237 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:14.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:13.135204+0000 mgr.smithi047.zgysuz (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:14.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:14 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:13.233256+0000 mon.smithi047 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.190:0/1617737847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:14.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:14.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:15.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.797+0000 7fde0dd8a700 1 -- 172.21.15.190:0/2622624444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 msgr2=0x7fde080f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:15.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.797+0000 7fde0dd8a700 1 --2- 172.21.15.190:0/2622624444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 0x7fde080f8f20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fddfc009fb0 tx=0x7fddfc009b30).stop 2023-11-15T04:37:15.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.797+0000 7fde0dd8a700 1 -- 172.21.15.190:0/2622624444 shutdown_connections 2023-11-15T04:37:15.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.797+0000 7fde0dd8a700 1 --2- 172.21.15.190:0/2622624444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 0x7fde080f8f20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:15.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.797+0000 7fde0dd8a700 1 -- 172.21.15.190:0/2622624444 >> 172.21.15.190:0/2622624444 conn(0x7fde080f09a0 msgr2=0x7fde080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:15.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.798+0000 7fde0dd8a700 1 -- 172.21.15.190:0/2622624444 shutdown_connections 2023-11-15T04:37:15.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.798+0000 7fde0dd8a700 1 -- 172.21.15.190:0/2622624444 wait complete. 2023-11-15T04:37:15.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.798+0000 7fde0dd8a700 1 Processor -- start 2023-11-15T04:37:15.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.798+0000 7fde0dd8a700 1 -- start start 2023-11-15T04:37:15.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.798+0000 7fde0dd8a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 0x7fde08106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:15.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.798+0000 7fde0dd8a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde081069f0 con 0x7fde080601e0 2023-11-15T04:37:15.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde0cd88700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 0x7fde08106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:15.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde0cd88700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 0x7fde08106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:52948/0 (socket says 172.21.15.190:52948) 2023-11-15T04:37:15.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde0cd88700 1 -- 172.21.15.190:0/3208295982 learned_addr learned my addr 172.21.15.190:0/3208295982 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:15.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde0cd88700 1 -- 172.21.15.190:0/3208295982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc005040 con 0x7fde080601e0 2023-11-15T04:37:15.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde0cd88700 1 --2- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 0x7fde08106500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fddfc013510 tx=0x7fddfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:15.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde057fa700 1 -- 172.21.15.190:0/3208295982 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddfc009320 con 0x7fde080601e0 2023-11-15T04:37:15.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde08106bf0 con 0x7fde080601e0 2023-11-15T04:37:15.808 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde057fa700 1 -- 172.21.15.190:0/3208295982 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddfc013f00 con 0x7fde080601e0 2023-11-15T04:37:15.808 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde057fa700 1 -- 172.21.15.190:0/3208295982 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddfc014d90 con 0x7fde080601e0 2023-11-15T04:37:15.808 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.799+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde08107080 con 0x7fde080601e0 2023-11-15T04:37:15.809 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.800+0000 7fde057fa700 1 -- 172.21.15.190:0/3208295982 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fddfc0095f0 con 0x7fde080601e0 2023-11-15T04:37:15.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.800+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde08045ad0 con 0x7fde080601e0 2023-11-15T04:37:15.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.800+0000 7fde057fa700 1 --2- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fddf802ff60 0x7fddf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:15.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.800+0000 7fde057fa700 1 -- 172.21.15.190:0/3208295982 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fddfc0410a0 con 0x7fde080601e0 2023-11-15T04:37:15.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.801+0000 7fde07fff700 1 --2- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fddf802ff60 0x7fddf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:15.811 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.801+0000 7fde07fff700 1 --2- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fddf802ff60 0x7fddf8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fddf40096a0 tx=0x7fddf4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:15.812 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:15.805+0000 7fde057fa700 1 -- 172.21.15.190:0/3208295982 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddfc01d7f0 con 0x7fde080601e0 2023-11-15T04:37:16.012 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.011+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde08107f30 con 0x7fde080601e0 2023-11-15T04:37:16.013 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.012+0000 7fde057fa700 1 -- 172.21.15.190:0/3208295982 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fddfc010230 con 0x7fde080601e0 2023-11-15T04:37:16.013 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:16.014 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:16.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fddf802ff60 msgr2=0x7fddf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:16.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 --2- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fddf802ff60 0x7fddf8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fddf40096a0 tx=0x7fddf4006b40).stop 2023-11-15T04:37:16.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 msgr2=0x7fde08106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:16.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 --2- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 0x7fde08106500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fddfc013510 tx=0x7fddfc00b040).stop 2023-11-15T04:37:16.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 shutdown_connections 2023-11-15T04:37:16.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 --2- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fddf802ff60 0x7fddf8032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:16.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 --2- 172.21.15.190:0/3208295982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde080601e0 0x7fde08106500 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:16.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 >> 172.21.15.190:0/3208295982 conn(0x7fde080f09a0 msgr2=0x7fde080f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:16.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 shutdown_connections 2023-11-15T04:37:16.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:16.014+0000 7fde0dd8a700 1 -- 172.21.15.190:0/3208295982 wait complete. 2023-11-15T04:37:16.017 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:15.135557+0000 mgr.smithi047.zgysuz (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:16.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:16 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:16.013652+0000 mon.smithi047 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.190:0/3208295982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:17.381 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:17.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:18.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:17 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:17.135937+0000 mgr.smithi047.zgysuz (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:18.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.533+0000 7f2e539f4700 1 -- 172.21.15.190:0/23669456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 msgr2=0x7f2e4c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:18.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.533+0000 7f2e539f4700 1 --2- 172.21.15.190:0/23669456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 0x7f2e4c0f4b60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f2e3c009fb0 tx=0x7f2e3c009b30).stop 2023-11-15T04:37:18.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.533+0000 7f2e539f4700 1 -- 172.21.15.190:0/23669456 shutdown_connections 2023-11-15T04:37:18.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.533+0000 7f2e539f4700 1 --2- 172.21.15.190:0/23669456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 0x7f2e4c0f4b60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:18.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.533+0000 7f2e539f4700 1 -- 172.21.15.190:0/23669456 >> 172.21.15.190:0/23669456 conn(0x7f2e4c0f0260 msgr2=0x7f2e4c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:18.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.533+0000 7f2e539f4700 1 -- 172.21.15.190:0/23669456 shutdown_connections 2023-11-15T04:37:18.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.533+0000 7f2e539f4700 1 -- 172.21.15.190:0/23669456 wait complete. 2023-11-15T04:37:18.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.534+0000 7f2e539f4700 1 Processor -- start 2023-11-15T04:37:18.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.534+0000 7f2e539f4700 1 -- start start 2023-11-15T04:37:18.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.534+0000 7f2e539f4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 0x7f2e4c0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:18.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.534+0000 7f2e539f4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c0f5770 con 0x7f2e4c0f4740 2023-11-15T04:37:18.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.534+0000 7f2e51790700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 0x7f2e4c0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:18.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.534+0000 7f2e51790700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 0x7f2e4c0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:52964/0 (socket says 172.21.15.190:52964) 2023-11-15T04:37:18.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.534+0000 7f2e51790700 1 -- 172.21.15.190:0/748584147 learned_addr learned my addr 172.21.15.190:0/748584147 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:18.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.535+0000 7f2e51790700 1 -- 172.21.15.190:0/748584147 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c005040 con 0x7f2e4c0f4740 2023-11-15T04:37:18.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.535+0000 7f2e51790700 1 --2- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 0x7f2e4c0f72f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f2e3c013510 tx=0x7f2e3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:18.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.535+0000 7f2e41ffb700 1 -- 172.21.15.190:0/748584147 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e3c009320 con 0x7f2e4c0f4740 2023-11-15T04:37:18.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.535+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4c0f5910 con 0x7f2e4c0f4740 2023-11-15T04:37:18.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.535+0000 7f2e41ffb700 1 -- 172.21.15.190:0/748584147 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e3c013f00 con 0x7f2e4c0f4740 2023-11-15T04:37:18.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.535+0000 7f2e41ffb700 1 -- 172.21.15.190:0/748584147 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e3c014d90 con 0x7f2e4c0f4740 2023-11-15T04:37:18.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.535+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c0f5da0 con 0x7f2e4c0f4740 2023-11-15T04:37:18.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.536+0000 7f2e41ffb700 1 -- 172.21.15.190:0/748584147 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e3c014420 con 0x7f2e4c0f4740 2023-11-15T04:37:18.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.536+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c059100 con 0x7f2e4c0f4740 2023-11-15T04:37:18.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.536+0000 7f2e41ffb700 1 --2- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e3802ff60 0x7f2e38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:18.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.536+0000 7f2e41ffb700 1 -- 172.21.15.190:0/748584147 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e3c041050 con 0x7f2e4c0f4740 2023-11-15T04:37:18.541 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.536+0000 7f2e50f8f700 1 --2- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e3802ff60 0x7f2e38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:18.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.537+0000 7f2e50f8f700 1 --2- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e3802ff60 0x7f2e38032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f2e480096a0 tx=0x7f2e48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:18.542 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.541+0000 7f2e41ffb700 1 -- 172.21.15.190:0/748584147 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e3c0095f0 con 0x7f2e4c0f4740 2023-11-15T04:37:18.746 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.745+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e4c045ad0 con 0x7f2e4c0f4740 2023-11-15T04:37:18.746 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.746+0000 7f2e41ffb700 1 -- 172.21.15.190:0/748584147 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e3c01db70 con 0x7f2e4c0f4740 2023-11-15T04:37:18.747 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:18.747 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:18.748 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e3802ff60 msgr2=0x7f2e38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:18.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 --2- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e3802ff60 0x7f2e38032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f2e480096a0 tx=0x7f2e48006b40).stop 2023-11-15T04:37:18.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 msgr2=0x7f2e4c0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:18.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 --2- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 0x7f2e4c0f72f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f2e3c013510 tx=0x7f2e3c00b040).stop 2023-11-15T04:37:18.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 shutdown_connections 2023-11-15T04:37:18.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 --2- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f2e3802ff60 0x7f2e38032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:18.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 --2- 172.21.15.190:0/748584147 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0f4740 0x7f2e4c0f72f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:18.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 >> 172.21.15.190:0/748584147 conn(0x7f2e4c0f0260 msgr2=0x7f2e4c0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:18.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 shutdown_connections 2023-11-15T04:37:18.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:18.747+0000 7f2e539f4700 1 -- 172.21.15.190:0/748584147 wait complete. 2023-11-15T04:37:18.751 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:19.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:18 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:18.747166+0000 mon.smithi047 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.190:0/748584147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:20.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:20.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:20.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:19 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:19.136158+0000 mgr.smithi047.zgysuz (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:21.320 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.318+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/2762308057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 msgr2=0x7fa3a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:21.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.318+0000 7fa3a6fe1700 1 --2- 172.21.15.190:0/2762308057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 0x7fa3a00f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fa394009fb0 tx=0x7fa394009b30).stop 2023-11-15T04:37:21.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.318+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/2762308057 shutdown_connections 2023-11-15T04:37:21.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.318+0000 7fa3a6fe1700 1 --2- 172.21.15.190:0/2762308057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 0x7fa3a00f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:21.321 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.318+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/2762308057 >> 172.21.15.190:0/2762308057 conn(0x7fa3a00f0b70 msgr2=0x7fa3a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:21.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.318+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/2762308057 shutdown_connections 2023-11-15T04:37:21.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.318+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/2762308057 wait complete. 2023-11-15T04:37:21.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.318+0000 7fa3a6fe1700 1 Processor -- start 2023-11-15T04:37:21.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.318+0000 7fa3a6fe1700 1 -- start start 2023-11-15T04:37:21.322 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.319+0000 7fa3a6fe1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 0x7fa3a010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.319+0000 7fa3a6fe1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0107240 con 0x7fa3a00f5410 2023-11-15T04:37:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.319+0000 7fa3a4d7d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 0x7fa3a010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.319+0000 7fa3a4d7d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 0x7fa3a010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:37252/0 (socket says 172.21.15.190:37252) 2023-11-15T04:37:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.319+0000 7fa3a4d7d700 1 -- 172.21.15.190:0/4032521770 learned_addr learned my addr 172.21.15.190:0/4032521770 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.319+0000 7fa3a4d7d700 1 -- 172.21.15.190:0/4032521770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa394005040 con 0x7fa3a00f5410 2023-11-15T04:37:21.324 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.320+0000 7fa3a4d7d700 1 --2- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 0x7fa3a010b850 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa394013510 tx=0x7fa39400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:21.324 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.320+0000 7fa39d7fa700 1 -- 172.21.15.190:0/4032521770 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa394009320 con 0x7fa3a00f5410 2023-11-15T04:37:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.320+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a0060220 con 0x7fa3a00f5410 2023-11-15T04:37:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.320+0000 7fa39d7fa700 1 -- 172.21.15.190:0/4032521770 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa394013f00 con 0x7fa3a00f5410 2023-11-15T04:37:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.320+0000 7fa39d7fa700 1 -- 172.21.15.190:0/4032521770 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa394014d90 con 0x7fa3a00f5410 2023-11-15T04:37:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.320+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a010bd40 con 0x7fa3a00f5410 2023-11-15T04:37:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.321+0000 7fa39d7fa700 1 -- 172.21.15.190:0/4032521770 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3940095f0 con 0x7fa3a00f5410 2023-11-15T04:37:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.321+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a0045ad0 con 0x7fa3a00f5410 2023-11-15T04:37:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.321+0000 7fa39d7fa700 1 --2- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39002ff60 0x7fa390032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:21.327 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.321+0000 7fa39d7fa700 1 -- 172.21.15.190:0/4032521770 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa394040fe0 con 0x7fa3a00f5410 2023-11-15T04:37:21.327 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.321+0000 7fa39ffff700 1 --2- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39002ff60 0x7fa390032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:21.327 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.321+0000 7fa39ffff700 1 --2- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39002ff60 0x7fa390032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa38c0096a0 tx=0x7fa38c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:21.327 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.325+0000 7fa39d7fa700 1 -- 172.21.15.190:0/4032521770 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa394010230 con 0x7fa3a00f5410 2023-11-15T04:37:21.535 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.534+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3a00fae60 con 0x7fa3a00f5410 2023-11-15T04:37:21.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.535+0000 7fa39d7fa700 1 -- 172.21.15.190:0/4032521770 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa394010230 con 0x7fa3a00f5410 2023-11-15T04:37:21.536 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:21.537 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:21.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39002ff60 msgr2=0x7fa390032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:21.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 --2- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39002ff60 0x7fa390032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa38c0096a0 tx=0x7fa38c006b40).stop 2023-11-15T04:37:21.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 msgr2=0x7fa3a010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:21.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 --2- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 0x7fa3a010b850 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa394013510 tx=0x7fa39400b040).stop 2023-11-15T04:37:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 shutdown_connections 2023-11-15T04:37:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 --2- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa39002ff60 0x7fa390032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 --2- 172.21.15.190:0/4032521770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3a00f5410 0x7fa3a010b850 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 >> 172.21.15.190:0/4032521770 conn(0x7fa3a00f0b70 msgr2=0x7fa3a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:21.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 shutdown_connections 2023-11-15T04:37:21.540 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:21.537+0000 7fa3a6fe1700 1 -- 172.21.15.190:0/4032521770 wait complete. 2023-11-15T04:37:21.540 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:21.136510+0000 mgr.smithi047.zgysuz (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:22.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:22 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:21.536523+0000 mon.smithi047 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.190:0/4032521770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:22.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:22.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:24.051 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.049+0000 7f9433e4f700 1 -- 172.21.15.190:0/1768575673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 msgr2=0x7f942c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:24.051 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.049+0000 7f9433e4f700 1 --2- 172.21.15.190:0/1768575673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 0x7f942c0f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f9420009fb0 tx=0x7f9420009b30).stop 2023-11-15T04:37:24.051 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.049+0000 7f9433e4f700 1 -- 172.21.15.190:0/1768575673 shutdown_connections 2023-11-15T04:37:24.052 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.049+0000 7f9433e4f700 1 --2- 172.21.15.190:0/1768575673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 0x7f942c0f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:24.052 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.049+0000 7f9433e4f700 1 -- 172.21.15.190:0/1768575673 >> 172.21.15.190:0/1768575673 conn(0x7f942c0f0b70 msgr2=0x7f942c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:24.052 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.049+0000 7f9433e4f700 1 -- 172.21.15.190:0/1768575673 shutdown_connections 2023-11-15T04:37:24.052 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.049+0000 7f9433e4f700 1 -- 172.21.15.190:0/1768575673 wait complete. 2023-11-15T04:37:24.052 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.050+0000 7f9433e4f700 1 Processor -- start 2023-11-15T04:37:24.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.050+0000 7f9433e4f700 1 -- start start 2023-11-15T04:37:24.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.050+0000 7f9433e4f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 0x7f942c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:24.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.050+0000 7f9433e4f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c105b20 con 0x7f942c0f5410 2023-11-15T04:37:24.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.050+0000 7f9431beb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 0x7f942c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:24.053 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.050+0000 7f9431beb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 0x7f942c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:37272/0 (socket says 172.21.15.190:37272) 2023-11-15T04:37:24.054 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.050+0000 7f9431beb700 1 -- 172.21.15.190:0/2006233930 learned_addr learned my addr 172.21.15.190:0/2006233930 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:24.055 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.051+0000 7f9431beb700 1 -- 172.21.15.190:0/2006233930 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420005040 con 0x7f942c0f5410 2023-11-15T04:37:24.055 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.051+0000 7f9431beb700 1 --2- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 0x7f942c105630 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f9420013510 tx=0x7f942000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:24.055 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.051+0000 7f941e7fc700 1 -- 172.21.15.190:0/2006233930 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9420009320 con 0x7f942c0f5410 2023-11-15T04:37:24.056 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.051+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942c105d20 con 0x7f942c0f5410 2023-11-15T04:37:24.056 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.051+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942c1061b0 con 0x7f942c0f5410 2023-11-15T04:37:24.056 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.051+0000 7f941e7fc700 1 -- 172.21.15.190:0/2006233930 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9420013f00 con 0x7f942c0f5410 2023-11-15T04:37:24.056 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.051+0000 7f941e7fc700 1 -- 172.21.15.190:0/2006233930 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9420014d90 con 0x7f942c0f5410 2023-11-15T04:37:24.056 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.052+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f942c045ad0 con 0x7f942c0f5410 2023-11-15T04:37:24.057 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.052+0000 7f941e7fc700 1 -- 172.21.15.190:0/2006233930 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94200095f0 con 0x7f942c0f5410 2023-11-15T04:37:24.057 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.052+0000 7f941e7fc700 1 --2- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f941802ff60 0x7f9418032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:24.057 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.052+0000 7f941e7fc700 1 -- 172.21.15.190:0/2006233930 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9420041000 con 0x7f942c0f5410 2023-11-15T04:37:24.057 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.052+0000 7f94313ea700 1 --2- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f941802ff60 0x7f9418032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:24.057 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.052+0000 7f94313ea700 1 --2- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f941802ff60 0x7f9418032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f94280096a0 tx=0x7f9428006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:24.058 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.057+0000 7f941e7fc700 1 -- 172.21.15.190:0/2006233930 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9420014420 con 0x7f942c0f5410 2023-11-15T04:37:24.263 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.262+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f942c0fae60 con 0x7f942c0f5410 2023-11-15T04:37:24.264 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.263+0000 7f941e7fc700 1 -- 172.21.15.190:0/2006233930 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9420010230 con 0x7f942c0f5410 2023-11-15T04:37:24.264 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:24.264 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:24.265 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f941802ff60 msgr2=0x7f9418032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:24.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 --2- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f941802ff60 0x7f9418032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f94280096a0 tx=0x7f9428006b40).stop 2023-11-15T04:37:24.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 msgr2=0x7f942c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:24.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 --2- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 0x7f942c105630 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f9420013510 tx=0x7f942000b040).stop 2023-11-15T04:37:24.266 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 shutdown_connections 2023-11-15T04:37:24.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 --2- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f941802ff60 0x7f9418032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:24.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 --2- 172.21.15.190:0/2006233930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f942c0f5410 0x7f942c105630 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:24.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 >> 172.21.15.190:0/2006233930 conn(0x7f942c0f0b70 msgr2=0x7f942c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:24.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 shutdown_connections 2023-11-15T04:37:24.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:24.265+0000 7f9433e4f700 1 -- 172.21.15.190:0/2006233930 wait complete. 2023-11-15T04:37:24.268 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:24.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:24 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:23.136814+0000 mgr.smithi047.zgysuz (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:25.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:25 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:24.264510+0000 mon.smithi047 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.190:0/2006233930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:25.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:25.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:26.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:26 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:25.137164+0000 mgr.smithi047.zgysuz (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:26.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.796+0000 7ff805dba700 1 -- 172.21.15.190:0/3899596399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 msgr2=0x7ff8000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:26.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.796+0000 7ff805dba700 1 --2- 172.21.15.190:0/3899596399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 0x7ff8000f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7ff7f0009fb0 tx=0x7ff7f0009b30).stop 2023-11-15T04:37:26.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.796+0000 7ff805dba700 1 -- 172.21.15.190:0/3899596399 shutdown_connections 2023-11-15T04:37:26.799 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.796+0000 7ff805dba700 1 --2- 172.21.15.190:0/3899596399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 0x7ff8000f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.796+0000 7ff805dba700 1 -- 172.21.15.190:0/3899596399 >> 172.21.15.190:0/3899596399 conn(0x7ff8000f0b70 msgr2=0x7ff8000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.796+0000 7ff805dba700 1 -- 172.21.15.190:0/3899596399 shutdown_connections 2023-11-15T04:37:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.796+0000 7ff805dba700 1 -- 172.21.15.190:0/3899596399 wait complete. 2023-11-15T04:37:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.797+0000 7ff805dba700 1 Processor -- start 2023-11-15T04:37:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.797+0000 7ff805dba700 1 -- start start 2023-11-15T04:37:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.797+0000 7ff805dba700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 0x7ff800109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.797+0000 7ff805dba700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff800060240 con 0x7ff8000f5410 2023-11-15T04:37:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.797+0000 7ff7ff7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 0x7ff800109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.797+0000 7ff7ff7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 0x7ff800109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:37282/0 (socket says 172.21.15.190:37282) 2023-11-15T04:37:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.797+0000 7ff7ff7fe700 1 -- 172.21.15.190:0/3070305658 learned_addr learned my addr 172.21.15.190:0/3070305658 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:26.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.798+0000 7ff7ff7fe700 1 -- 172.21.15.190:0/3070305658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f0005040 con 0x7ff8000f5410 2023-11-15T04:37:26.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.798+0000 7ff7ff7fe700 1 --2- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 0x7ff800109430 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7ff7f0013510 tx=0x7ff7f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:26.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.798+0000 7ff7f7fff700 1 -- 172.21.15.190:0/3070305658 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f0009320 con 0x7ff8000f5410 2023-11-15T04:37:26.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.798+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff800060440 con 0x7ff8000f5410 2023-11-15T04:37:26.802 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.798+0000 7ff7f7fff700 1 -- 172.21.15.190:0/3070305658 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7f0013f00 con 0x7ff8000f5410 2023-11-15T04:37:26.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.798+0000 7ff7f7fff700 1 -- 172.21.15.190:0/3070305658 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f0014d90 con 0x7ff8000f5410 2023-11-15T04:37:26.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.798+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff800109ae0 con 0x7ff8000f5410 2023-11-15T04:37:26.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.799+0000 7ff7f7fff700 1 -- 172.21.15.190:0/3070305658 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7f00095f0 con 0x7ff8000f5410 2023-11-15T04:37:26.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.799+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff800045ad0 con 0x7ff8000f5410 2023-11-15T04:37:26.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.799+0000 7ff7f7fff700 1 --2- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff7ec02ff60 0x7ff7ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:26.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.799+0000 7ff7f7fff700 1 -- 172.21.15.190:0/3070305658 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7f00410a0 con 0x7ff8000f5410 2023-11-15T04:37:26.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.799+0000 7ff7feffd700 1 --2- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff7ec02ff60 0x7ff7ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:26.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.800+0000 7ff7feffd700 1 --2- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff7ec02ff60 0x7ff7ec032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff7e80096a0 tx=0x7ff7e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:26.805 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:26.804+0000 7ff7f7fff700 1 -- 172.21.15.190:0/3070305658 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7f001d7f0 con 0x7ff8000f5410 2023-11-15T04:37:27.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.010+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8000fae60 con 0x7ff8000f5410 2023-11-15T04:37:27.011 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.010+0000 7ff7f7fff700 1 -- 172.21.15.190:0/3070305658 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7f0010230 con 0x7ff8000f5410 2023-11-15T04:37:27.012 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:27.012 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:27.013 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff7ec02ff60 msgr2=0x7ff7ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:27.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 --2- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff7ec02ff60 0x7ff7ec032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff7e80096a0 tx=0x7ff7e8006b40).stop 2023-11-15T04:37:27.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 msgr2=0x7ff800109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:27.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 --2- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 0x7ff800109430 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7ff7f0013510 tx=0x7ff7f000b040).stop 2023-11-15T04:37:27.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 shutdown_connections 2023-11-15T04:37:27.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 --2- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff7ec02ff60 0x7ff7ec032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:27.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 --2- 172.21.15.190:0/3070305658 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8000f5410 0x7ff800109430 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:27.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 >> 172.21.15.190:0/3070305658 conn(0x7ff8000f0b70 msgr2=0x7ff8000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:27.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 shutdown_connections 2023-11-15T04:37:27.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:27.012+0000 7ff805dba700 1 -- 172.21.15.190:0/3070305658 wait complete. 2023-11-15T04:37:27.015 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:28.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:27.011728+0000 mon.smithi047 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.190:0/3070305658' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:28.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:27 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:27.137373+0000 mgr.smithi047.zgysuz (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:28.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:28.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:29.588 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.587+0000 7f73f5a8e700 1 -- 172.21.15.190:0/3303797890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 msgr2=0x7f73f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:29.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.587+0000 7f73f5a8e700 1 --2- 172.21.15.190:0/3303797890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 0x7f73f00f90f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f73d8009fb0 tx=0x7f73d8009b30).stop 2023-11-15T04:37:29.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.587+0000 7f73f5a8e700 1 -- 172.21.15.190:0/3303797890 shutdown_connections 2023-11-15T04:37:29.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.587+0000 7f73f5a8e700 1 --2- 172.21.15.190:0/3303797890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 0x7f73f00f90f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:29.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.587+0000 7f73f5a8e700 1 -- 172.21.15.190:0/3303797890 >> 172.21.15.190:0/3303797890 conn(0x7f73f00f0b70 msgr2=0x7f73f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:29.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.587+0000 7f73f5a8e700 1 -- 172.21.15.190:0/3303797890 shutdown_connections 2023-11-15T04:37:29.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.587+0000 7f73f5a8e700 1 -- 172.21.15.190:0/3303797890 wait complete. 2023-11-15T04:37:29.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.588+0000 7f73f5a8e700 1 Processor -- start 2023-11-15T04:37:29.590 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.588+0000 7f73f5a8e700 1 -- start start 2023-11-15T04:37:29.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.588+0000 7f73f5a8e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 0x7f73f01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:29.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.588+0000 7f73f5a8e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f0106bd0 con 0x7f73f0060280 2023-11-15T04:37:29.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.588+0000 7f73ef7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 0x7f73f01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:29.592 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.588+0000 7f73ef7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 0x7f73f01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:37306/0 (socket says 172.21.15.190:37306) 2023-11-15T04:37:29.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.588+0000 7f73ef7fe700 1 -- 172.21.15.190:0/1064682367 learned_addr learned my addr 172.21.15.190:0/1064682367 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:29.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.589+0000 7f73ef7fe700 1 -- 172.21.15.190:0/1064682367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d8005040 con 0x7f73f0060280 2023-11-15T04:37:29.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.589+0000 7f73ef7fe700 1 --2- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 0x7f73f01066e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f73d8013510 tx=0x7f73d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:29.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.589+0000 7f73e7fff700 1 -- 172.21.15.190:0/1064682367 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73d8009320 con 0x7f73f0060280 2023-11-15T04:37:29.593 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.589+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f0106dd0 con 0x7f73f0060280 2023-11-15T04:37:29.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.589+0000 7f73e7fff700 1 -- 172.21.15.190:0/1064682367 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73d8013f00 con 0x7f73f0060280 2023-11-15T04:37:29.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.589+0000 7f73e7fff700 1 -- 172.21.15.190:0/1064682367 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73d8014dc0 con 0x7f73f0060280 2023-11-15T04:37:29.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.589+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f0107260 con 0x7f73f0060280 2023-11-15T04:37:29.594 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.590+0000 7f73e7fff700 1 -- 172.21.15.190:0/1064682367 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73d80095f0 con 0x7f73f0060280 2023-11-15T04:37:29.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.590+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f0045ad0 con 0x7f73f0060280 2023-11-15T04:37:29.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.590+0000 7f73e7fff700 1 --2- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73dc02ff60 0x7f73dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:29.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.590+0000 7f73e7fff700 1 -- 172.21.15.190:0/1064682367 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f73d8041030 con 0x7f73f0060280 2023-11-15T04:37:29.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.590+0000 7f73eeffd700 1 --2- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73dc02ff60 0x7f73dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:29.595 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.590+0000 7f73eeffd700 1 --2- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73dc02ff60 0x7f73dc032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f73e00096a0 tx=0x7f73e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:29.596 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.595+0000 7f73e7fff700 1 -- 172.21.15.190:0/1064682367 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73d8014420 con 0x7f73f0060280 2023-11-15T04:37:29.803 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.802+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73f0108110 con 0x7f73f0060280 2023-11-15T04:37:29.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.804+0000 7f73e7fff700 1 -- 172.21.15.190:0/1064682367 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f73d8010230 con 0x7f73f0060280 2023-11-15T04:37:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:29.805 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:29.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73dc02ff60 msgr2=0x7f73dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:29.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 --2- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73dc02ff60 0x7f73dc032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f73e00096a0 tx=0x7f73e0006b40).stop 2023-11-15T04:37:29.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 msgr2=0x7f73f01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:29.806 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 --2- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 0x7f73f01066e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f73d8013510 tx=0x7f73d800b040).stop 2023-11-15T04:37:29.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 shutdown_connections 2023-11-15T04:37:29.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 --2- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f73dc02ff60 0x7f73dc032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:29.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 --2- 172.21.15.190:0/1064682367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73f0060280 0x7f73f01066e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:29.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 >> 172.21.15.190:0/1064682367 conn(0x7f73f00f0b70 msgr2=0x7f73f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:29.807 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 shutdown_connections 2023-11-15T04:37:29.808 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:29.805+0000 7f73f5a8e700 1 -- 172.21.15.190:0/1064682367 wait complete. 2023-11-15T04:37:29.808 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:30.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:29.137661+0000 mgr.smithi047.zgysuz (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:30.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:30 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:29.804502+0000 mon.smithi047 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.190:0/1064682367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:31.194 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:31.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:32.347 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.345+0000 7ff76359e700 1 -- 172.21.15.190:0/1137901161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 msgr2=0x7ff7640f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:32.347 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.345+0000 7ff76359e700 1 --2- 172.21.15.190:0/1137901161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 0x7ff7640f8f80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7ff74c009fb0 tx=0x7ff74c009b30).stop 2023-11-15T04:37:32.347 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.345+0000 7ff76359e700 1 -- 172.21.15.190:0/1137901161 shutdown_connections 2023-11-15T04:37:32.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.345+0000 7ff76359e700 1 --2- 172.21.15.190:0/1137901161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 0x7ff7640f8f80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:32.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.345+0000 7ff76359e700 1 -- 172.21.15.190:0/1137901161 >> 172.21.15.190:0/1137901161 conn(0x7ff7640f09a0 msgr2=0x7ff7640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:32.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.345+0000 7ff76359e700 1 -- 172.21.15.190:0/1137901161 shutdown_connections 2023-11-15T04:37:32.348 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.345+0000 7ff76359e700 1 -- 172.21.15.190:0/1137901161 wait complete. 2023-11-15T04:37:32.349 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.345+0000 7ff76359e700 1 Processor -- start 2023-11-15T04:37:32.349 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.346+0000 7ff76359e700 1 -- start start 2023-11-15T04:37:32.349 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.346+0000 7ff76359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 0x7ff764106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:32.349 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.346+0000 7ff76359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7641069f0 con 0x7ff764060180 2023-11-15T04:37:32.349 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.346+0000 7ff76259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 0x7ff764106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:32.350 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.346+0000 7ff76259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 0x7ff764106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:58306/0 (socket says 172.21.15.190:58306) 2023-11-15T04:37:32.350 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.346+0000 7ff76259c700 1 -- 172.21.15.190:0/2398076547 learned_addr learned my addr 172.21.15.190:0/2398076547 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:32.350 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.346+0000 7ff76259c700 1 -- 172.21.15.190:0/2398076547 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74c005040 con 0x7ff764060180 2023-11-15T04:37:32.352 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.346+0000 7ff76259c700 1 --2- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 0x7ff764106500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7ff74c014000 tx=0x7ff74c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:32.352 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.347+0000 7ff75affd700 1 -- 172.21.15.190:0/2398076547 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74c0095f0 con 0x7ff764060180 2023-11-15T04:37:32.352 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.347+0000 7ff75affd700 1 -- 172.21.15.190:0/2398076547 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff74c007e70 con 0x7ff764060180 2023-11-15T04:37:32.352 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.347+0000 7ff75affd700 1 -- 172.21.15.190:0/2398076547 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74c014ea0 con 0x7ff764060180 2023-11-15T04:37:32.353 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.347+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff764106bf0 con 0x7ff764060180 2023-11-15T04:37:32.353 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.347+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff764107080 con 0x7ff764060180 2023-11-15T04:37:32.353 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.347+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff764045ad0 con 0x7ff764060180 2023-11-15T04:37:32.353 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.348+0000 7ff75affd700 1 -- 172.21.15.190:0/2398076547 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff74c025070 con 0x7ff764060180 2023-11-15T04:37:32.354 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.348+0000 7ff75affd700 1 --2- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff75002ff60 0x7ff750032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:32.354 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.348+0000 7ff75affd700 1 -- 172.21.15.190:0/2398076547 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff74c040a40 con 0x7ff764060180 2023-11-15T04:37:32.354 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.348+0000 7ff761d9b700 1 --2- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff75002ff60 0x7ff750032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:32.354 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.348+0000 7ff761d9b700 1 --2- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff75002ff60 0x7ff750032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff7540096a0 tx=0x7ff754006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:32.355 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.352+0000 7ff75affd700 1 -- 172.21.15.190:0/2398076547 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff74c0079c0 con 0x7ff764060180 2023-11-15T04:37:32.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:32 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:31.137929+0000 mgr.smithi047.zgysuz (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:32.556 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.555+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff764107f30 con 0x7ff764060180 2023-11-15T04:37:32.557 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.556+0000 7ff75affd700 1 -- 172.21.15.190:0/2398076547 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff74c00f300 con 0x7ff764060180 2023-11-15T04:37:32.558 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:32.558 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:32.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff75002ff60 msgr2=0x7ff750032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:32.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 --2- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff75002ff60 0x7ff750032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff7540096a0 tx=0x7ff754006b40).stop 2023-11-15T04:37:32.559 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 msgr2=0x7ff764106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:32.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 --2- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 0x7ff764106500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7ff74c014000 tx=0x7ff74c00b040).stop 2023-11-15T04:37:32.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 shutdown_connections 2023-11-15T04:37:32.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 --2- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7ff75002ff60 0x7ff750032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:32.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 --2- 172.21.15.190:0/2398076547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff764060180 0x7ff764106500 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:32.560 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 >> 172.21.15.190:0/2398076547 conn(0x7ff7640f09a0 msgr2=0x7ff7640f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:32.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 shutdown_connections 2023-11-15T04:37:32.561 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:32.558+0000 7ff76359e700 1 -- 172.21.15.190:0/2398076547 wait complete. 2023-11-15T04:37:32.561 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:33.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:33 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:32.557351+0000 mon.smithi047 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.190:0/2398076547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:33.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:33.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:34.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:34 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:33.138259+0000 mgr.smithi047.zgysuz (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:35.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.103+0000 7fbd8e777700 1 -- 172.21.15.190:0/1143603452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 msgr2=0x7fbd880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:35.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.103+0000 7fbd8e777700 1 --2- 172.21.15.190:0/1143603452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 0x7fbd880f55c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fbd84009fb0 tx=0x7fbd84009b30).stop 2023-11-15T04:37:35.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.103+0000 7fbd8e777700 1 -- 172.21.15.190:0/1143603452 shutdown_connections 2023-11-15T04:37:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.103+0000 7fbd8e777700 1 --2- 172.21.15.190:0/1143603452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 0x7fbd880f55c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.103+0000 7fbd8e777700 1 -- 172.21.15.190:0/1143603452 >> 172.21.15.190:0/1143603452 conn(0x7fbd880f09a0 msgr2=0x7fbd880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.103+0000 7fbd8e777700 1 -- 172.21.15.190:0/1143603452 shutdown_connections 2023-11-15T04:37:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.103+0000 7fbd8e777700 1 -- 172.21.15.190:0/1143603452 wait complete. 2023-11-15T04:37:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.103+0000 7fbd8e777700 1 Processor -- start 2023-11-15T04:37:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.103+0000 7fbd8e777700 1 -- start start 2023-11-15T04:37:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.104+0000 7fbd8e777700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 0x7fbd881091a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.104+0000 7fbd8e777700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88060280 con 0x7fbd880f51a0 2023-11-15T04:37:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.104+0000 7fbd8d775700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 0x7fbd881091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.104+0000 7fbd8d775700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 0x7fbd881091a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:58322/0 (socket says 172.21.15.190:58322) 2023-11-15T04:37:35.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.104+0000 7fbd8d775700 1 -- 172.21.15.190:0/645536974 learned_addr learned my addr 172.21.15.190:0/645536974 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:35.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.104+0000 7fbd8d775700 1 -- 172.21.15.190:0/645536974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd84005040 con 0x7fbd880f51a0 2023-11-15T04:37:35.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.104+0000 7fbd8d775700 1 --2- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 0x7fbd881091a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fbd84013510 tx=0x7fbd8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:35.109 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.105+0000 7fbd7dffb700 1 -- 172.21.15.190:0/645536974 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd84009320 con 0x7fbd880f51a0 2023-11-15T04:37:35.109 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.105+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd88060480 con 0x7fbd880f51a0 2023-11-15T04:37:35.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.105+0000 7fbd7dffb700 1 -- 172.21.15.190:0/645536974 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd84013f00 con 0x7fbd880f51a0 2023-11-15T04:37:35.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.105+0000 7fbd7dffb700 1 -- 172.21.15.190:0/645536974 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd84014dc0 con 0x7fbd880f51a0 2023-11-15T04:37:35.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.105+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd881098d0 con 0x7fbd880f51a0 2023-11-15T04:37:35.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.106+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd88045ad0 con 0x7fbd880f51a0 2023-11-15T04:37:35.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.106+0000 7fbd7dffb700 1 -- 172.21.15.190:0/645536974 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd840095f0 con 0x7fbd880f51a0 2023-11-15T04:37:35.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.106+0000 7fbd7dffb700 1 --2- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbd7402ff60 0x7fbd74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:35.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.106+0000 7fbd7dffb700 1 -- 172.21.15.190:0/645536974 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd84041040 con 0x7fbd880f51a0 2023-11-15T04:37:35.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.106+0000 7fbd8cf74700 1 --2- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbd7402ff60 0x7fbd74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:35.111 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.107+0000 7fbd8cf74700 1 --2- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbd7402ff60 0x7fbd74032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fbd780096a0 tx=0x7fbd78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:35.112 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.111+0000 7fbd7dffb700 1 -- 172.21.15.190:0/645536974 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd84014420 con 0x7fbd880f51a0 2023-11-15T04:37:35.316 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.314+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd880fabe0 con 0x7fbd880f51a0 2023-11-15T04:37:35.316 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.315+0000 7fbd7dffb700 1 -- 172.21.15.190:0/645536974 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd84010230 con 0x7fbd880f51a0 2023-11-15T04:37:35.317 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:35.317 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:35.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbd7402ff60 msgr2=0x7fbd74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:35.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 --2- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbd7402ff60 0x7fbd74032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fbd780096a0 tx=0x7fbd78006b40).stop 2023-11-15T04:37:35.318 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 msgr2=0x7fbd881091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:35.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 --2- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 0x7fbd881091a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fbd84013510 tx=0x7fbd8400b040).stop 2023-11-15T04:37:35.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 shutdown_connections 2023-11-15T04:37:35.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 --2- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fbd7402ff60 0x7fbd74032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:35.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 --2- 172.21.15.190:0/645536974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd880f51a0 0x7fbd881091a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:35.319 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 >> 172.21.15.190:0/645536974 conn(0x7fbd880f09a0 msgr2=0x7fbd880f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:35.320 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 shutdown_connections 2023-11-15T04:37:35.320 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:35.317+0000 7fbd8e777700 1 -- 172.21.15.190:0/645536974 wait complete. 2023-11-15T04:37:35.320 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:35.138512+0000 mgr.smithi047.zgysuz (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:36.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:36 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:35.316680+0000 mon.smithi047 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.190:0/645536974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:36.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:36.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:37.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 -- 172.21.15.190:0/3886732693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 msgr2=0x7fa0e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:37.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 --2- 172.21.15.190:0/3886732693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 0x7fa0e00f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fa0dc009fb0 tx=0x7fa0dc009b30).stop 2023-11-15T04:37:37.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 -- 172.21.15.190:0/3886732693 shutdown_connections 2023-11-15T04:37:37.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 --2- 172.21.15.190:0/3886732693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 0x7fa0e00f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:37.913 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 -- 172.21.15.190:0/3886732693 >> 172.21.15.190:0/3886732693 conn(0x7fa0e00f1c70 msgr2=0x7fa0e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:37.913 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 -- 172.21.15.190:0/3886732693 shutdown_connections 2023-11-15T04:37:37.913 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 -- 172.21.15.190:0/3886732693 wait complete. 2023-11-15T04:37:37.913 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 Processor -- start 2023-11-15T04:37:37.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 -- start start 2023-11-15T04:37:37.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 0x7fa0e0108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:37.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.910+0000 7fa0e7d36700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e0108df0 con 0x7fa0e00f6510 2023-11-15T04:37:37.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0e5ad2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 0x7fa0e0108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:37.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0e5ad2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 0x7fa0e0108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:58340/0 (socket says 172.21.15.190:58340) 2023-11-15T04:37:37.915 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0e5ad2700 1 -- 172.21.15.190:0/4047781263 learned_addr learned my addr 172.21.15.190:0/4047781263 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:37.915 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0e5ad2700 1 -- 172.21.15.190:0/4047781263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc005040 con 0x7fa0e00f6510 2023-11-15T04:37:37.915 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0e5ad2700 1 --2- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 0x7fa0e0108900 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fa0dc013510 tx=0x7fa0dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:37.917 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0d67fc700 1 -- 172.21.15.190:0/4047781263 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0dc009320 con 0x7fa0e00f6510 2023-11-15T04:37:37.918 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e0108ff0 con 0x7fa0e00f6510 2023-11-15T04:37:37.918 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0d67fc700 1 -- 172.21.15.190:0/4047781263 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0dc013f00 con 0x7fa0e00f6510 2023-11-15T04:37:37.918 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0d67fc700 1 -- 172.21.15.190:0/4047781263 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0dc014d90 con 0x7fa0e00f6510 2023-11-15T04:37:37.918 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.911+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e0109480 con 0x7fa0e00f6510 2023-11-15T04:37:37.919 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.912+0000 7fa0d67fc700 1 -- 172.21.15.190:0/4047781263 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0dc0095f0 con 0x7fa0e00f6510 2023-11-15T04:37:37.919 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.912+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e0045ad0 con 0x7fa0e00f6510 2023-11-15T04:37:37.919 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.912+0000 7fa0d67fc700 1 --2- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa0cc02ff60 0x7fa0cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:37.919 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.912+0000 7fa0d67fc700 1 -- 172.21.15.190:0/4047781263 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0dc0410e0 con 0x7fa0e00f6510 2023-11-15T04:37:37.919 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.913+0000 7fa0e52d1700 1 --2- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa0cc02ff60 0x7fa0cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.913+0000 7fa0e52d1700 1 --2- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa0cc02ff60 0x7fa0cc032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa0d00096a0 tx=0x7fa0d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:37.918+0000 7fa0d67fc700 1 -- 172.21.15.190:0/4047781263 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0dc014420 con 0x7fa0e00f6510 2023-11-15T04:37:38.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.123+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0e010a330 con 0x7fa0e00f6510 2023-11-15T04:37:38.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.124+0000 7fa0d67fc700 1 -- 172.21.15.190:0/4047781263 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa0dc01d7f0 con 0x7fa0e00f6510 2023-11-15T04:37:38.125 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:38.125 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:38.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa0cc02ff60 msgr2=0x7fa0cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:38.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 --2- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa0cc02ff60 0x7fa0cc032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa0d00096a0 tx=0x7fa0d0006b40).stop 2023-11-15T04:37:38.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 msgr2=0x7fa0e0108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:38.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 --2- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 0x7fa0e0108900 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fa0dc013510 tx=0x7fa0dc00b040).stop 2023-11-15T04:37:38.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 shutdown_connections 2023-11-15T04:37:38.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 --2- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fa0cc02ff60 0x7fa0cc032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:38.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 --2- 172.21.15.190:0/4047781263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0e00f6510 0x7fa0e0108900 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:38.128 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 >> 172.21.15.190:0/4047781263 conn(0x7fa0e00f1c70 msgr2=0x7fa0e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:38.128 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 shutdown_connections 2023-11-15T04:37:38.128 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:38.126+0000 7fa0e7d36700 1 -- 172.21.15.190:0/4047781263 wait complete. 2023-11-15T04:37:38.128 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:38.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:37 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:37.138863+0000 mgr.smithi047.zgysuz (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:39.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:38 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:38.125137+0000 mon.smithi047 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.190:0/4047781263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:39.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:39.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:40.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:39 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:39.139257+0000 mgr.smithi047.zgysuz (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:40.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.682+0000 7f09ba913700 1 -- 172.21.15.190:0/3862675961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 msgr2=0x7f09b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:40.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.682+0000 7f09ba913700 1 --2- 172.21.15.190:0/3862675961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 0x7f09b40f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f09a4009fb0 tx=0x7f09a4009b30).stop 2023-11-15T04:37:40.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.682+0000 7f09ba913700 1 -- 172.21.15.190:0/3862675961 shutdown_connections 2023-11-15T04:37:40.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.682+0000 7f09ba913700 1 --2- 172.21.15.190:0/3862675961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 0x7f09b40f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:40.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.683+0000 7f09ba913700 1 -- 172.21.15.190:0/3862675961 >> 172.21.15.190:0/3862675961 conn(0x7f09b40f0b70 msgr2=0x7f09b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:40.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.683+0000 7f09ba913700 1 -- 172.21.15.190:0/3862675961 shutdown_connections 2023-11-15T04:37:40.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.683+0000 7f09ba913700 1 -- 172.21.15.190:0/3862675961 wait complete. 2023-11-15T04:37:40.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.683+0000 7f09ba913700 1 Processor -- start 2023-11-15T04:37:40.686 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.683+0000 7f09ba913700 1 -- start start 2023-11-15T04:37:40.687 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.684+0000 7f09ba913700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 0x7f09b4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:40.687 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.684+0000 7f09ba913700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b4105b20 con 0x7f09b40f5410 2023-11-15T04:37:40.687 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.684+0000 7f09b3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 0x7f09b4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:40.687 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.684+0000 7f09b3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 0x7f09b4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60154/0 (socket says 172.21.15.190:60154) 2023-11-15T04:37:40.687 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.684+0000 7f09b3fff700 1 -- 172.21.15.190:0/51525246 learned_addr learned my addr 172.21.15.190:0/51525246 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:40.688 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.684+0000 7f09b3fff700 1 -- 172.21.15.190:0/51525246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4005040 con 0x7f09b40f5410 2023-11-15T04:37:40.688 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.684+0000 7f09b3fff700 1 --2- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 0x7f09b4105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f09a4013510 tx=0x7f09a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:40.688 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.685+0000 7f09b0ff9700 1 -- 172.21.15.190:0/51525246 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a4009320 con 0x7f09b40f5410 2023-11-15T04:37:40.688 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.685+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b4105d20 con 0x7f09b40f5410 2023-11-15T04:37:40.688 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.685+0000 7f09b0ff9700 1 -- 172.21.15.190:0/51525246 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09a4013f00 con 0x7f09b40f5410 2023-11-15T04:37:40.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.685+0000 7f09b0ff9700 1 -- 172.21.15.190:0/51525246 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a4014d90 con 0x7f09b40f5410 2023-11-15T04:37:40.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.685+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b41061b0 con 0x7f09b40f5410 2023-11-15T04:37:40.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.685+0000 7f09b0ff9700 1 -- 172.21.15.190:0/51525246 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09a40263a0 con 0x7f09b40f5410 2023-11-15T04:37:40.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.686+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b4045ad0 con 0x7f09b40f5410 2023-11-15T04:37:40.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.686+0000 7f09b0ff9700 1 --2- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:40.689 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.686+0000 7f09b0ff9700 1 -- 172.21.15.190:0/51525246 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09a4041340 con 0x7f09b40f5410 2023-11-15T04:37:40.690 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.686+0000 7f09b37fe700 1 --2- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:40.690 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.686+0000 7f09b37fe700 1 --2- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f099c02ff60 0x7f099c032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f09a80096a0 tx=0x7f09a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:40.690 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.691+0000 7f09b0ff9700 1 -- 172.21.15.190:0/51525246 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09a40126f0 con 0x7f09b40f5410 2023-11-15T04:37:40.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.898+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09b40fae60 con 0x7f09b40f5410 2023-11-15T04:37:40.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.899+0000 7f09b0ff9700 1 -- 172.21.15.190:0/51525246 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09a4012330 con 0x7f09b40f5410 2023-11-15T04:37:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:40.899 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:40.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f099c02ff60 msgr2=0x7f099c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:40.900 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 --2- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f099c02ff60 0x7f099c032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f09a80096a0 tx=0x7f09a8006b40).stop 2023-11-15T04:37:40.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 msgr2=0x7f09b4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:40.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 --2- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 0x7f09b4105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f09a4013510 tx=0x7f09a400b040).stop 2023-11-15T04:37:40.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 shutdown_connections 2023-11-15T04:37:40.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 --2- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:40.901 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 --2- 172.21.15.190:0/51525246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09b40f5410 0x7f09b4105630 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:40.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 >> 172.21.15.190:0/51525246 conn(0x7f09b40f0b70 msgr2=0x7f09b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:40.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 shutdown_connections 2023-11-15T04:37:40.902 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:40.900+0000 7f09ba913700 1 -- 172.21.15.190:0/51525246 wait complete. 2023-11-15T04:37:40.902 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:41.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:40 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:40.899056+0000 mon.smithi047 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.190:0/51525246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:42.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:41 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:41.139599+0000 mgr.smithi047.zgysuz (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:42.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:42.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:43.407 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.407+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/2489282767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 msgr2=0x7f0dd40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:43.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.407+0000 7f0ddb2e6700 1 --2- 172.21.15.190:0/2489282767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 0x7f0dd40f6930 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f0dc4009fb0 tx=0x7f0dc4009b30).stop 2023-11-15T04:37:43.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.407+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/2489282767 shutdown_connections 2023-11-15T04:37:43.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.407+0000 7f0ddb2e6700 1 --2- 172.21.15.190:0/2489282767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 0x7f0dd40f6930 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:43.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.407+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/2489282767 >> 172.21.15.190:0/2489282767 conn(0x7f0dd40f1c70 msgr2=0x7f0dd40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:43.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.407+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/2489282767 shutdown_connections 2023-11-15T04:37:43.409 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.407+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/2489282767 wait complete. 2023-11-15T04:37:43.409 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.408+0000 7f0ddb2e6700 1 Processor -- start 2023-11-15T04:37:43.409 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.408+0000 7f0ddb2e6700 1 -- start start 2023-11-15T04:37:43.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.408+0000 7f0ddb2e6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 0x7f0dd4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:43.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.408+0000 7f0ddb2e6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4106c20 con 0x7f0dd40f6510 2023-11-15T04:37:43.412 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.408+0000 7f0dd9082700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 0x7f0dd4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:43.412 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.408+0000 7f0dd9082700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 0x7f0dd4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60168/0 (socket says 172.21.15.190:60168) 2023-11-15T04:37:43.412 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.408+0000 7f0dd9082700 1 -- 172.21.15.190:0/3612858169 learned_addr learned my addr 172.21.15.190:0/3612858169 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:43.412 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.408+0000 7f0dd9082700 1 -- 172.21.15.190:0/3612858169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4005040 con 0x7f0dd40f6510 2023-11-15T04:37:43.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.409+0000 7f0dd9082700 1 --2- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 0x7f0dd4106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f0dc4013510 tx=0x7f0dc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:43.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.409+0000 7f0dc9ffb700 1 -- 172.21.15.190:0/3612858169 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dc4009320 con 0x7f0dd40f6510 2023-11-15T04:37:43.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.409+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd4106e20 con 0x7f0dd40f6510 2023-11-15T04:37:43.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.409+0000 7f0dc9ffb700 1 -- 172.21.15.190:0/3612858169 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dc4013f00 con 0x7f0dd40f6510 2023-11-15T04:37:43.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.409+0000 7f0dc9ffb700 1 -- 172.21.15.190:0/3612858169 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dc4014d90 con 0x7f0dd40f6510 2023-11-15T04:37:43.414 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.409+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd41072b0 con 0x7f0dd40f6510 2023-11-15T04:37:43.414 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.410+0000 7f0dc9ffb700 1 -- 172.21.15.190:0/3612858169 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0dc40095f0 con 0x7f0dd40f6510 2023-11-15T04:37:43.414 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.410+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd4045ad0 con 0x7f0dd40f6510 2023-11-15T04:37:43.414 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.410+0000 7f0dc9ffb700 1 --2- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0dc002ff60 0x7f0dc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:43.414 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.410+0000 7f0dc9ffb700 1 -- 172.21.15.190:0/3612858169 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0dc40410c0 con 0x7f0dd40f6510 2023-11-15T04:37:43.415 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.410+0000 7f0dd8881700 1 --2- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0dc002ff60 0x7f0dc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:43.415 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.411+0000 7f0dd8881700 1 --2- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0dc002ff60 0x7f0dc0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0dd00096a0 tx=0x7f0dd0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:43.415 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.415+0000 7f0dc9ffb700 1 -- 172.21.15.190:0/3612858169 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0dc401d7f0 con 0x7f0dd40f6510 2023-11-15T04:37:43.623 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.623+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0dd40fbf60 con 0x7f0dd40f6510 2023-11-15T04:37:43.624 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.624+0000 7f0dc9ffb700 1 -- 172.21.15.190:0/3612858169 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0dc4010230 con 0x7f0dd40f6510 2023-11-15T04:37:43.624 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:43.625 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:43.626 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0dc002ff60 msgr2=0x7f0dc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:43.627 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 --2- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0dc002ff60 0x7f0dc0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0dd00096a0 tx=0x7f0dd0006b40).stop 2023-11-15T04:37:43.627 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 msgr2=0x7f0dd4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:43.627 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 --2- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 0x7f0dd4106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f0dc4013510 tx=0x7f0dc400b040).stop 2023-11-15T04:37:43.627 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 shutdown_connections 2023-11-15T04:37:43.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 --2- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f0dc002ff60 0x7f0dc0032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:43.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 --2- 172.21.15.190:0/3612858169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dd40f6510 0x7f0dd4106730 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:43.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 >> 172.21.15.190:0/3612858169 conn(0x7f0dd40f1c70 msgr2=0x7f0dd40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:43.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 shutdown_connections 2023-11-15T04:37:43.628 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:43.626+0000 7f0ddb2e6700 1 -- 172.21.15.190:0/3612858169 wait complete. 2023-11-15T04:37:43.628 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:43.139866+0000 mgr.smithi047.zgysuz (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:44.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:44 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:43.624625+0000 mon.smithi047 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.190:0/3612858169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:45.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:45.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.180+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/3088927256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 msgr2=0x7fb6ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:46.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.180+0000 7fb6f2a9a700 1 --2- 172.21.15.190:0/3088927256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 0x7fb6ec0f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fb6e0009fb0 tx=0x7fb6e0009b30).stop 2023-11-15T04:37:46.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.180+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/3088927256 shutdown_connections 2023-11-15T04:37:46.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.180+0000 7fb6f2a9a700 1 --2- 172.21.15.190:0/3088927256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 0x7fb6ec0f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:46.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.180+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/3088927256 >> 172.21.15.190:0/3088927256 conn(0x7fb6ec0f0b70 msgr2=0x7fb6ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:46.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.180+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/3088927256 shutdown_connections 2023-11-15T04:37:46.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.180+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/3088927256 wait complete. 2023-11-15T04:37:46.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.181+0000 7fb6f2a9a700 1 Processor -- start 2023-11-15T04:37:46.183 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.181+0000 7fb6f2a9a700 1 -- start start 2023-11-15T04:37:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.181+0000 7fb6f2a9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 0x7fb6ec109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.181+0000 7fb6f2a9a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec060240 con 0x7fb6ec0f5410 2023-11-15T04:37:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.181+0000 7fb6f0836700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 0x7fb6ec109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.181+0000 7fb6f0836700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 0x7fb6ec109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60190/0 (socket says 172.21.15.190:60190) 2023-11-15T04:37:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.181+0000 7fb6f0836700 1 -- 172.21.15.190:0/944111164 learned_addr learned my addr 172.21.15.190:0/944111164 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.182+0000 7fb6f0836700 1 -- 172.21.15.190:0/944111164 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e0005040 con 0x7fb6ec0f5410 2023-11-15T04:37:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.182+0000 7fb6f0836700 1 --2- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 0x7fb6ec109430 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fb6e0013510 tx=0x7fb6e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.182+0000 7fb6e97fa700 1 -- 172.21.15.190:0/944111164 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6e0009320 con 0x7fb6ec0f5410 2023-11-15T04:37:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.182+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec060440 con 0x7fb6ec0f5410 2023-11-15T04:37:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.182+0000 7fb6e97fa700 1 -- 172.21.15.190:0/944111164 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6e0013f00 con 0x7fb6ec0f5410 2023-11-15T04:37:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.182+0000 7fb6e97fa700 1 -- 172.21.15.190:0/944111164 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6e0014d90 con 0x7fb6ec0f5410 2023-11-15T04:37:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.182+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ec109ae0 con 0x7fb6ec0f5410 2023-11-15T04:37:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.183+0000 7fb6e97fa700 1 -- 172.21.15.190:0/944111164 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6e00095f0 con 0x7fb6ec0f5410 2023-11-15T04:37:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.183+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ec045ad0 con 0x7fb6ec0f5410 2023-11-15T04:37:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.183+0000 7fb6e97fa700 1 --2- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb6dc02ff60 0x7fb6dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.183+0000 7fb6e97fa700 1 -- 172.21.15.190:0/944111164 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6e0040fe0 con 0x7fb6ec0f5410 2023-11-15T04:37:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.183+0000 7fb6ebfff700 1 --2- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb6dc02ff60 0x7fb6dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.184+0000 7fb6ebfff700 1 --2- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb6dc02ff60 0x7fb6dc032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb6d80096a0 tx=0x7fb6d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.188+0000 7fb6e97fa700 1 -- 172.21.15.190:0/944111164 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6e001d7f0 con 0x7fb6ec0f5410 2023-11-15T04:37:46.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.393+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6ec0fae60 con 0x7fb6ec0f5410 2023-11-15T04:37:46.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.394+0000 7fb6e97fa700 1 -- 172.21.15.190:0/944111164 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb6e0010230 con 0x7fb6ec0f5410 2023-11-15T04:37:46.394 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:46.394 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb6dc02ff60 msgr2=0x7fb6dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 --2- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb6dc02ff60 0x7fb6dc032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb6d80096a0 tx=0x7fb6d8006b40).stop 2023-11-15T04:37:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 msgr2=0x7fb6ec109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 --2- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 0x7fb6ec109430 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fb6e0013510 tx=0x7fb6e000b040).stop 2023-11-15T04:37:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 shutdown_connections 2023-11-15T04:37:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 --2- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb6dc02ff60 0x7fb6dc032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 --2- 172.21.15.190:0/944111164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6ec0f5410 0x7fb6ec109430 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 >> 172.21.15.190:0/944111164 conn(0x7fb6ec0f0b70 msgr2=0x7fb6ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 shutdown_connections 2023-11-15T04:37:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:46.395+0000 7fb6f2a9a700 1 -- 172.21.15.190:0/944111164 wait complete. 2023-11-15T04:37:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:46.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:46 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:45.140170+0000 mgr.smithi047.zgysuz (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:47.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:47 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:46.394146+0000 mon.smithi047 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.190:0/944111164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:47.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:47.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:48.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:48 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:47.140513+0000 mgr.smithi047.zgysuz (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:48.924 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.922+0000 7fb907dd4700 1 -- 172.21.15.190:0/2177883404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 msgr2=0x7fb9000f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:48.924 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.922+0000 7fb907dd4700 1 --2- 172.21.15.190:0/2177883404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 0x7fb9000f90f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fb8f0009fb0 tx=0x7fb8f0009b30).stop 2023-11-15T04:37:48.924 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.922+0000 7fb907dd4700 1 -- 172.21.15.190:0/2177883404 shutdown_connections 2023-11-15T04:37:48.924 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.922+0000 7fb907dd4700 1 --2- 172.21.15.190:0/2177883404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 0x7fb9000f90f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:48.925 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.922+0000 7fb907dd4700 1 -- 172.21.15.190:0/2177883404 >> 172.21.15.190:0/2177883404 conn(0x7fb9000f0b70 msgr2=0x7fb9000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:48.925 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.922+0000 7fb907dd4700 1 -- 172.21.15.190:0/2177883404 shutdown_connections 2023-11-15T04:37:48.925 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.922+0000 7fb907dd4700 1 -- 172.21.15.190:0/2177883404 wait complete. 2023-11-15T04:37:48.925 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.923+0000 7fb907dd4700 1 Processor -- start 2023-11-15T04:37:48.925 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.923+0000 7fb907dd4700 1 -- start start 2023-11-15T04:37:48.926 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.923+0000 7fb907dd4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 0x7fb9001066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:48.926 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.923+0000 7fb907dd4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb900106bd0 con 0x7fb900060280 2023-11-15T04:37:48.926 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.923+0000 7fb905b70700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 0x7fb9001066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:48.926 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.923+0000 7fb905b70700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 0x7fb9001066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:60206/0 (socket says 172.21.15.190:60206) 2023-11-15T04:37:48.926 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.923+0000 7fb905b70700 1 -- 172.21.15.190:0/4114191592 learned_addr learned my addr 172.21.15.190:0/4114191592 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:48.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.924+0000 7fb905b70700 1 -- 172.21.15.190:0/4114191592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0005040 con 0x7fb900060280 2023-11-15T04:37:48.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.924+0000 7fb905b70700 1 --2- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 0x7fb9001066e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fb8f0009830 tx=0x7fb8f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:48.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.924+0000 7fb8f67fc700 1 -- 172.21.15.190:0/4114191592 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8f0009320 con 0x7fb900060280 2023-11-15T04:37:48.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.924+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb900106dd0 con 0x7fb900060280 2023-11-15T04:37:48.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.924+0000 7fb8f67fc700 1 -- 172.21.15.190:0/4114191592 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8f00136c0 con 0x7fb900060280 2023-11-15T04:37:48.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.924+0000 7fb8f67fc700 1 -- 172.21.15.190:0/4114191592 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8f001ee90 con 0x7fb900060280 2023-11-15T04:37:48.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.924+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb900107260 con 0x7fb900060280 2023-11-15T04:37:48.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.925+0000 7fb8f67fc700 1 -- 172.21.15.190:0/4114191592 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8f001e420 con 0x7fb900060280 2023-11-15T04:37:48.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.925+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb900045ad0 con 0x7fb900060280 2023-11-15T04:37:48.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.925+0000 7fb8f67fc700 1 --2- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8ec02ff10 0x7fb8ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:48.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.925+0000 7fb8f67fc700 1 -- 172.21.15.190:0/4114191592 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8f0040f30 con 0x7fb900060280 2023-11-15T04:37:48.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.925+0000 7fb90536f700 1 --2- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8ec02ff10 0x7fb8ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:48.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.925+0000 7fb90536f700 1 --2- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8ec02ff10 0x7fb8ec0323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fb8fc0096a0 tx=0x7fb8fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:48.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:48.930+0000 7fb8f67fc700 1 -- 172.21.15.190:0/4114191592 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8f006d2b0 con 0x7fb900060280 2023-11-15T04:37:49.135 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.135+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb900108110 con 0x7fb900060280 2023-11-15T04:37:49.135 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.135+0000 7fb8f67fc700 1 -- 172.21.15.190:0/4114191592 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8f001c790 con 0x7fb900060280 2023-11-15T04:37:49.136 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:49.136 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:49.137 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8ec02ff10 msgr2=0x7fb8ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:49.137 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 --2- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8ec02ff10 0x7fb8ec0323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fb8fc0096a0 tx=0x7fb8fc006b40).stop 2023-11-15T04:37:49.137 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 msgr2=0x7fb9001066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:49.138 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 --2- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 0x7fb9001066e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fb8f0009830 tx=0x7fb8f000b040).stop 2023-11-15T04:37:49.138 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 shutdown_connections 2023-11-15T04:37:49.138 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 --2- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fb8ec02ff10 0x7fb8ec0323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:49.138 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 --2- 172.21.15.190:0/4114191592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb900060280 0x7fb9001066e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:49.138 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 >> 172.21.15.190:0/4114191592 conn(0x7fb9000f0b70 msgr2=0x7fb9000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:49.139 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 shutdown_connections 2023-11-15T04:37:49.139 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:49.137+0000 7fb907dd4700 1 -- 172.21.15.190:0/4114191592 wait complete. 2023-11-15T04:37:49.139 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:49.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:49 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:49.135972+0000 mon.smithi047 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.190:0/4114191592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:50.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:50 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:49.140875+0000 mgr.smithi047.zgysuz (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:50.525 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:50.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:51.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.676+0000 7f8f40151700 1 -- 172.21.15.190:0/3975523877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 msgr2=0x7f8f380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:51.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.676+0000 7f8f40151700 1 --2- 172.21.15.190:0/3975523877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 0x7f8f380f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f8f34009fb0 tx=0x7f8f34009b30).stop 2023-11-15T04:37:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.676+0000 7f8f40151700 1 -- 172.21.15.190:0/3975523877 shutdown_connections 2023-11-15T04:37:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.676+0000 7f8f40151700 1 --2- 172.21.15.190:0/3975523877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 0x7f8f380f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.676+0000 7f8f40151700 1 -- 172.21.15.190:0/3975523877 >> 172.21.15.190:0/3975523877 conn(0x7f8f380f0b70 msgr2=0x7f8f380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.676+0000 7f8f40151700 1 -- 172.21.15.190:0/3975523877 shutdown_connections 2023-11-15T04:37:51.678 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.676+0000 7f8f40151700 1 -- 172.21.15.190:0/3975523877 wait complete. 2023-11-15T04:37:51.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.677+0000 7f8f40151700 1 Processor -- start 2023-11-15T04:37:51.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.677+0000 7f8f40151700 1 -- start start 2023-11-15T04:37:51.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.677+0000 7f8f40151700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 0x7f8f38109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:51.680 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.677+0000 7f8f40151700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f38060220 con 0x7f8f380f5410 2023-11-15T04:37:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.677+0000 7f8f3deed700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 0x7f8f38109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.677+0000 7f8f3deed700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 0x7f8f38109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:55686/0 (socket says 172.21.15.190:55686) 2023-11-15T04:37:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.677+0000 7f8f3deed700 1 -- 172.21.15.190:0/3571209699 learned_addr learned my addr 172.21.15.190:0/3571209699 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.677+0000 7f8f3deed700 1 -- 172.21.15.190:0/3571209699 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f34005040 con 0x7f8f380f5410 2023-11-15T04:37:51.682 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.677+0000 7f8f3deed700 1 --2- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 0x7f8f38109450 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f8f34013510 tx=0x7f8f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:51.682 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.678+0000 7f8f2e7fc700 1 -- 172.21.15.190:0/3571209699 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f34009320 con 0x7f8f380f5410 2023-11-15T04:37:51.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.678+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f38060420 con 0x7f8f380f5410 2023-11-15T04:37:51.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.678+0000 7f8f2e7fc700 1 -- 172.21.15.190:0/3571209699 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f34013f00 con 0x7f8f380f5410 2023-11-15T04:37:51.683 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.678+0000 7f8f2e7fc700 1 -- 172.21.15.190:0/3571209699 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f34014d90 con 0x7f8f380f5410 2023-11-15T04:37:51.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.678+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f38109b00 con 0x7f8f380f5410 2023-11-15T04:37:51.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.679+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f38059100 con 0x7f8f380f5410 2023-11-15T04:37:51.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.679+0000 7f8f2e7fc700 1 -- 172.21.15.190:0/3571209699 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f340095f0 con 0x7f8f380f5410 2023-11-15T04:37:51.684 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.679+0000 7f8f2e7fc700 1 --2- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f2402ff60 0x7f8f24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:51.685 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.679+0000 7f8f2e7fc700 1 -- 172.21.15.190:0/3571209699 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f34041030 con 0x7f8f380f5410 2023-11-15T04:37:51.685 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.679+0000 7f8f3d6ec700 1 --2- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f2402ff60 0x7f8f24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:51.685 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.679+0000 7f8f3d6ec700 1 --2- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f2402ff60 0x7f8f24032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8f280096a0 tx=0x7f8f28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:51.686 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.684+0000 7f8f2e7fc700 1 -- 172.21.15.190:0/3571209699 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f34014420 con 0x7f8f380f5410 2023-11-15T04:37:51.893 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.894+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f38045ad0 con 0x7f8f380f5410 2023-11-15T04:37:51.894 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.894+0000 7f8f2e7fc700 1 -- 172.21.15.190:0/3571209699 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f34010230 con 0x7f8f380f5410 2023-11-15T04:37:51.895 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:51.895 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:51.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f2402ff60 msgr2=0x7f8f24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:51.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 --2- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f2402ff60 0x7f8f24032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8f280096a0 tx=0x7f8f28006b40).stop 2023-11-15T04:37:51.896 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 msgr2=0x7f8f38109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:51.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 --2- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 0x7f8f38109450 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f8f34013510 tx=0x7f8f3400b040).stop 2023-11-15T04:37:51.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 shutdown_connections 2023-11-15T04:37:51.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 --2- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f8f2402ff60 0x7f8f24032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:51.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 --2- 172.21.15.190:0/3571209699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f380f5410 0x7f8f38109450 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:51.897 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 >> 172.21.15.190:0/3571209699 conn(0x7f8f380f0b70 msgr2=0x7f8f380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:51.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 shutdown_connections 2023-11-15T04:37:51.898 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:51.896+0000 7f8f40151700 1 -- 172.21.15.190:0/3571209699 wait complete. 2023-11-15T04:37:51.898 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:52.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:51.141201+0000 mgr.smithi047.zgysuz (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:52.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:52 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:51.894787+0000 mon.smithi047 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.190:0/3571209699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:53.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:53.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:54.446 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.445+0000 7f728ca44700 1 -- 172.21.15.190:0/1999326477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 msgr2=0x7f72880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:54.446 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.445+0000 7f728ca44700 1 --2- 172.21.15.190:0/1999326477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 0x7f72880f55c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f7278009fb0 tx=0x7f7278009b30).stop 2023-11-15T04:37:54.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.445+0000 7f728ca44700 1 -- 172.21.15.190:0/1999326477 shutdown_connections 2023-11-15T04:37:54.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.445+0000 7f728ca44700 1 --2- 172.21.15.190:0/1999326477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 0x7f72880f55c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:54.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.445+0000 7f728ca44700 1 -- 172.21.15.190:0/1999326477 >> 172.21.15.190:0/1999326477 conn(0x7f72880f09a0 msgr2=0x7f72880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:54.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.446+0000 7f728ca44700 1 -- 172.21.15.190:0/1999326477 shutdown_connections 2023-11-15T04:37:54.447 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.446+0000 7f728ca44700 1 -- 172.21.15.190:0/1999326477 wait complete. 2023-11-15T04:37:54.448 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.446+0000 7f728ca44700 1 Processor -- start 2023-11-15T04:37:54.448 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.446+0000 7f728ca44700 1 -- start start 2023-11-15T04:37:54.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.447+0000 7f728ca44700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 0x7f72881091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:54.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.447+0000 7f728ca44700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7288060240 con 0x7f72880f51a0 2023-11-15T04:37:54.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.447+0000 7f72877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 0x7f72881091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:54.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.447+0000 7f72877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 0x7f72881091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:55708/0 (socket says 172.21.15.190:55708) 2023-11-15T04:37:54.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.447+0000 7f72877fe700 1 -- 172.21.15.190:0/1202225183 learned_addr learned my addr 172.21.15.190:0/1202225183 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:54.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.447+0000 7f72877fe700 1 -- 172.21.15.190:0/1202225183 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7278005040 con 0x7f72880f51a0 2023-11-15T04:37:54.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.447+0000 7f72877fe700 1 --2- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 0x7f72881091b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f7278013510 tx=0x7f727800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:54.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.448+0000 7f726ffff700 1 -- 172.21.15.190:0/1202225183 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7278009320 con 0x7f72880f51a0 2023-11-15T04:37:54.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.448+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7288060440 con 0x7f72880f51a0 2023-11-15T04:37:54.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.448+0000 7f726ffff700 1 -- 172.21.15.190:0/1202225183 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7278013f00 con 0x7f72880f51a0 2023-11-15T04:37:54.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.448+0000 7f726ffff700 1 -- 172.21.15.190:0/1202225183 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7278014dc0 con 0x7f72880f51a0 2023-11-15T04:37:54.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.448+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7288109860 con 0x7f72880f51a0 2023-11-15T04:37:54.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.449+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7288045ad0 con 0x7f72880f51a0 2023-11-15T04:37:54.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.449+0000 7f726ffff700 1 -- 172.21.15.190:0/1202225183 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72780095f0 con 0x7f72880f51a0 2023-11-15T04:37:54.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.449+0000 7f726ffff700 1 --2- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f727002ff60 0x7f7270032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:54.453 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.449+0000 7f726ffff700 1 -- 172.21.15.190:0/1202225183 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72780410e0 con 0x7f72880f51a0 2023-11-15T04:37:54.453 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.449+0000 7f7286ffd700 1 --2- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f727002ff60 0x7f7270032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:54.453 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.449+0000 7f7286ffd700 1 --2- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f727002ff60 0x7f7270032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f727c0096a0 tx=0x7f727c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:54.454 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.454+0000 7f726ffff700 1 -- 172.21.15.190:0/1202225183 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7278014420 con 0x7f72880f51a0 2023-11-15T04:37:54.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:54 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:53.141545+0000 mgr.smithi047.zgysuz (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:54.660 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.660+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f728810a070 con 0x7f72880f51a0 2023-11-15T04:37:54.661 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.660+0000 7f726ffff700 1 -- 172.21.15.190:0/1202225183 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f727801c920 con 0x7f72880f51a0 2023-11-15T04:37:54.661 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:54.661 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:54.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f727002ff60 msgr2=0x7f7270032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:54.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 --2- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f727002ff60 0x7f7270032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f727c0096a0 tx=0x7f727c006b40).stop 2023-11-15T04:37:54.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 msgr2=0x7f72881091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:54.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 --2- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 0x7f72881091b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f7278013510 tx=0x7f727800b040).stop 2023-11-15T04:37:54.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 shutdown_connections 2023-11-15T04:37:54.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 --2- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f727002ff60 0x7f7270032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:54.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 --2- 172.21.15.190:0/1202225183 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72880f51a0 0x7f72881091b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:54.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 >> 172.21.15.190:0/1202225183 conn(0x7f72880f09a0 msgr2=0x7f72880f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:54.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 shutdown_connections 2023-11-15T04:37:54.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:54.662+0000 7f728ca44700 1 -- 172.21.15.190:0/1202225183 wait complete. 2023-11-15T04:37:54.664 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:55.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:55 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:54.660964+0000 mon.smithi047 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.190:0/1202225183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:56.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:56.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:56.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:56 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:55.141951+0000 mgr.smithi047.zgysuz (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:57.168 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.167+0000 7f174965a700 1 -- 172.21.15.190:0/609656874 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 msgr2=0x7f17440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:57.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.167+0000 7f174965a700 1 --2- 172.21.15.190:0/609656874 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 0x7f17440f6930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f1738009fb0 tx=0x7f1738009b30).stop 2023-11-15T04:37:57.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.167+0000 7f174965a700 1 -- 172.21.15.190:0/609656874 shutdown_connections 2023-11-15T04:37:57.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.167+0000 7f174965a700 1 --2- 172.21.15.190:0/609656874 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 0x7f17440f6930 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:57.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.167+0000 7f174965a700 1 -- 172.21.15.190:0/609656874 >> 172.21.15.190:0/609656874 conn(0x7f17440f1c70 msgr2=0x7f17440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:57.169 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.168+0000 7f174965a700 1 -- 172.21.15.190:0/609656874 shutdown_connections 2023-11-15T04:37:57.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.168+0000 7f174965a700 1 -- 172.21.15.190:0/609656874 wait complete. 2023-11-15T04:37:57.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.168+0000 7f174965a700 1 Processor -- start 2023-11-15T04:37:57.170 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.168+0000 7f174965a700 1 -- start start 2023-11-15T04:37:57.172 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.169+0000 7f174965a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 0x7f1744106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:57.172 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.169+0000 7f174965a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1744106c20 con 0x7f17440f6510 2023-11-15T04:37:57.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.169+0000 7f1742ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 0x7f1744106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:57.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.169+0000 7f1742ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 0x7f1744106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:55732/0 (socket says 172.21.15.190:55732) 2023-11-15T04:37:57.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.169+0000 7f1742ffd700 1 -- 172.21.15.190:0/372706192 learned_addr learned my addr 172.21.15.190:0/372706192 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:57.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.169+0000 7f1742ffd700 1 -- 172.21.15.190:0/372706192 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1738005040 con 0x7f17440f6510 2023-11-15T04:37:57.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.169+0000 7f1742ffd700 1 --2- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 0x7f1744106730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f1738013510 tx=0x7f173800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:57.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.170+0000 7f172b7fe700 1 -- 172.21.15.190:0/372706192 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1738009320 con 0x7f17440f6510 2023-11-15T04:37:57.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.170+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1744106e20 con 0x7f17440f6510 2023-11-15T04:37:57.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.170+0000 7f172b7fe700 1 -- 172.21.15.190:0/372706192 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1738013f00 con 0x7f17440f6510 2023-11-15T04:37:57.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.170+0000 7f172b7fe700 1 -- 172.21.15.190:0/372706192 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1738014d90 con 0x7f17440f6510 2023-11-15T04:37:57.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.170+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17441072b0 con 0x7f17440f6510 2023-11-15T04:37:57.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.171+0000 7f172b7fe700 1 -- 172.21.15.190:0/372706192 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17380263a0 con 0x7f17440f6510 2023-11-15T04:37:57.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.171+0000 7f172b7fe700 1 --2- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f172c02ff60 0x7f172c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:57.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.171+0000 7f172b7fe700 1 -- 172.21.15.190:0/372706192 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1738041000 con 0x7f17440f6510 2023-11-15T04:37:57.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.171+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1744045ad0 con 0x7f17440f6510 2023-11-15T04:37:57.176 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.171+0000 7f17427fc700 1 --2- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f172c02ff60 0x7f172c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:57.176 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.171+0000 7f17427fc700 1 --2- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f172c02ff60 0x7f172c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f17340096a0 tx=0x7f1734006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:57.176 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.176+0000 7f172b7fe700 1 -- 172.21.15.190:0/372706192 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1738012920 con 0x7f17440f6510 2023-11-15T04:37:57.380 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.380+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17440fbf60 con 0x7f17440f6510 2023-11-15T04:37:57.381 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.381+0000 7f172b7fe700 1 -- 172.21.15.190:0/372706192 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f173802b020 con 0x7f17440f6510 2023-11-15T04:37:57.382 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:37:57.382 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:37:57.383 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.382+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f172c02ff60 msgr2=0x7f172c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:57.383 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.382+0000 7f174965a700 1 --2- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f172c02ff60 0x7f172c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f17340096a0 tx=0x7f1734006b40).stop 2023-11-15T04:37:57.383 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.382+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 msgr2=0x7f1744106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:57.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.382+0000 7f174965a700 1 --2- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 0x7f1744106730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f1738013510 tx=0x7f173800b040).stop 2023-11-15T04:37:57.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.383+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 shutdown_connections 2023-11-15T04:37:57.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.383+0000 7f174965a700 1 --2- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7f172c02ff60 0x7f172c032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:57.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.383+0000 7f174965a700 1 --2- 172.21.15.190:0/372706192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17440f6510 0x7f1744106730 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:57.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.383+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 >> 172.21.15.190:0/372706192 conn(0x7f17440f1c70 msgr2=0x7f17440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:57.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.383+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 shutdown_connections 2023-11-15T04:37:57.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:57.383+0000 7f174965a700 1 -- 172.21.15.190:0/372706192 wait complete. 2023-11-15T04:37:57.385 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:37:58.257 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:57.142280+0000 mgr.smithi047.zgysuz (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:37:58.258 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:37:57 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:37:57.381458+0000 mon.smithi047 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.190:0/372706192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:37:58.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:37:58.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:37:59.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.907+0000 7fe2c254f700 1 -- 172.21.15.190:0/3494193587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 msgr2=0x7fe2bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:37:59.908 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.907+0000 7fe2c254f700 1 --2- 172.21.15.190:0/3494193587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 0x7fe2bc0f5830 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fe2b0009fb0 tx=0x7fe2b0009b30).stop 2023-11-15T04:37:59.908 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.907+0000 7fe2c254f700 1 -- 172.21.15.190:0/3494193587 shutdown_connections 2023-11-15T04:37:59.908 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.907+0000 7fe2c254f700 1 --2- 172.21.15.190:0/3494193587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 0x7fe2bc0f5830 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:37:59.908 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.907+0000 7fe2c254f700 1 -- 172.21.15.190:0/3494193587 >> 172.21.15.190:0/3494193587 conn(0x7fe2bc0f0b70 msgr2=0x7fe2bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:37:59.909 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.907+0000 7fe2c254f700 1 -- 172.21.15.190:0/3494193587 shutdown_connections 2023-11-15T04:37:59.909 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.907+0000 7fe2c254f700 1 -- 172.21.15.190:0/3494193587 wait complete. 2023-11-15T04:37:59.909 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.908+0000 7fe2c254f700 1 Processor -- start 2023-11-15T04:37:59.911 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.908+0000 7fe2c254f700 1 -- start start 2023-11-15T04:37:59.911 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.908+0000 7fe2c254f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 0x7fe2bc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:59.911 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.908+0000 7fe2c254f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc105b20 con 0x7fe2bc0f5410 2023-11-15T04:37:59.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.908+0000 7fe2bbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 0x7fe2bc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:59.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.908+0000 7fe2bbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 0x7fe2bc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.190:55740/0 (socket says 172.21.15.190:55740) 2023-11-15T04:37:59.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.908+0000 7fe2bbfff700 1 -- 172.21.15.190:0/3973140886 learned_addr learned my addr 172.21.15.190:0/3973140886 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-15T04:37:59.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.909+0000 7fe2bbfff700 1 -- 172.21.15.190:0/3973140886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b0005040 con 0x7fe2bc0f5410 2023-11-15T04:37:59.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.909+0000 7fe2bbfff700 1 --2- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 0x7fe2bc105630 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fe2b0013510 tx=0x7fe2b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:59.913 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.909+0000 7fe2b8ff9700 1 -- 172.21.15.190:0/3973140886 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2b0009320 con 0x7fe2bc0f5410 2023-11-15T04:37:59.913 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.909+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2bc105d20 con 0x7fe2bc0f5410 2023-11-15T04:37:59.913 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.909+0000 7fe2b8ff9700 1 -- 172.21.15.190:0/3973140886 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2b0013f00 con 0x7fe2bc0f5410 2023-11-15T04:37:59.913 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.909+0000 7fe2b8ff9700 1 -- 172.21.15.190:0/3973140886 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2b0014d90 con 0x7fe2bc0f5410 2023-11-15T04:37:59.913 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.909+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2bc1061b0 con 0x7fe2bc0f5410 2023-11-15T04:37:59.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.910+0000 7fe2b8ff9700 1 -- 172.21.15.190:0/3973140886 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2b0014420 con 0x7fe2bc0f5410 2023-11-15T04:37:59.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.910+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2bc045ad0 con 0x7fe2bc0f5410 2023-11-15T04:37:59.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.910+0000 7fe2b8ff9700 1 --2- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe2a402ffb0 0x7fe2a4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:37:59.914 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.910+0000 7fe2b8ff9700 1 -- 172.21.15.190:0/3973140886 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2b0040ff0 con 0x7fe2bc0f5410 2023-11-15T04:37:59.915 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.910+0000 7fe2bb7fe700 1 --2- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe2a402ffb0 0x7fe2a4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:37:59.915 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.911+0000 7fe2bb7fe700 1 --2- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe2a402ffb0 0x7fe2a4032470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe2ac0096a0 tx=0x7fe2ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:37:59.915 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:37:59.915+0000 7fe2b8ff9700 1 -- 172.21.15.190:0/3973140886 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2b006d2b0 con 0x7fe2bc0f5410 2023-11-15T04:38:00.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.123+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2bc0fae60 con 0x7fe2bc0f5410 2023-11-15T04:38:00.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.124+0000 7fe2b8ff9700 1 -- 172.21.15.190:0/3973140886 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe2b00095f0 con 0x7fe2bc0f5410 2023-11-15T04:38:00.124 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-15T04:38:00.124 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"4f7ad8e8-836f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:28:06.573922Z","created":"2023-11-15T04:28:06.573922Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:38:00.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.125+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe2a402ffb0 msgr2=0x7fe2a4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:38:00.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.126+0000 7fe2c254f700 1 --2- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe2a402ffb0 0x7fe2a4032470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe2ac0096a0 tx=0x7fe2ac006b40).stop 2023-11-15T04:38:00.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.126+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 msgr2=0x7fe2bc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:38:00.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.126+0000 7fe2c254f700 1 --2- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 0x7fe2bc105630 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fe2b0013510 tx=0x7fe2b000b040).stop 2023-11-15T04:38:00.126 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.126+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 shutdown_connections 2023-11-15T04:38:00.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.126+0000 7fe2c254f700 1 --2- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:6800/1025769174,v1:172.21.15.47:6801/1025769174] conn(0x7fe2a402ffb0 0x7fe2a4032470 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:38:00.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.126+0000 7fe2c254f700 1 --2- 172.21.15.190:0/3973140886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2bc0f5410 0x7fe2bc105630 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:38:00.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.126+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 >> 172.21.15.190:0/3973140886 conn(0x7fe2bc0f0b70 msgr2=0x7fe2bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:38:00.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.126+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 shutdown_connections 2023-11-15T04:38:00.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-15T04:38:00.126+0000 7fe2c254f700 1 -- 172.21.15.190:0/3973140886 wait complete. 2023-11-15T04:38:00.128 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-11-15T04:38:00.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:38:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: cluster 2023-11-15T04:37:59.142577+0000 mgr.smithi047.zgysuz (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:38:00.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:38:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: audit 2023-11-15T04:38:00.124284+0000 mon.smithi047 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.190:0/3973140886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:38:00.511 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-15T04:38:00.511 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-15T04:38:00.512 DEBUG:teuthology.orchestra.run.smithi047:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T04:38:00.537 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T04:38:00.555 INFO:tasks.cephadm:Stopping all daemons... 2023-11-15T04:38:00.556 INFO:tasks.cephadm.mon.smithi047:Stopping mon.smithi047... 2023-11-15T04:38:00.556 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047 2023-11-15T04:38:00.875 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:38:00 smithi047 systemd[1]: Stopping Ceph mon.smithi047 for 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7... 2023-11-15T04:38:00.875 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:38:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:38:00.693+0000 7fd967eb2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi047 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T04:38:00.875 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Nov 15 04:38:00 smithi047 ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7-mon.smithi047[112378]: debug 2023-11-15T04:38:00.693+0000 7fd967eb2700 -1 mon.smithi047@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T04:38:01.031 DEBUG:teuthology.orchestra.run.smithi047:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi047.service' 2023-11-15T04:38:01.094 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-15T04:38:01.095 INFO:tasks.cephadm.mon.smithi047:Stopped mon.smithi047 2023-11-15T04:38:01.095 INFO:tasks.cephadm.mon.smithi190:Stopping mon.smithi190... 2023-11-15T04:38:01.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi190 2023-11-15T04:38:01.128 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi190.service: Unit ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi190.service not loaded. 2023-11-15T04:38:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-15T04:38:01.131 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi190" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 5: 'sudo systemctl stop ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi190' 2023-11-15T04:38:01.131 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-15T04:38:01.133 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354/remote/ubuntu@smithi047.front.sepia.ceph.com/crash 2023-11-15T04:38:01.135 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T04:38:01.182 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354/remote/ubuntu@smithi190.front.sepia.ceph.com/crash 2023-11-15T04:38:01.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T04:38:01.226 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-15T04:38:01.227 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2023-11-15T04:38:01.227 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-15T04:38:01.228 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-15T04:38:01.266 INFO:teuthology.orchestra.run.smithi047.stderr:grep: /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-15T04:38:01.269 INFO:tasks.cephadm:Compressing logs... 2023-11-15T04:38:01.269 DEBUG:teuthology.orchestra.run.smithi047:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T04:38:01.312 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T04:38:01.339 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T04:38:01.339 INFO:teuthology.orchestra.run.smithi190.stderr:: No such file or directory 2023-11-15T04:38:01.341 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T04:38:01.342 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-11-15T04:38:01.343 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m0.013s 2023-11-15T04:38:01.343 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m0.008s 2023-11-15T04:38:01.343 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m0.013s 2023-11-15T04:38:01.344 INFO:teuthology.orchestra.run.smithi047.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-15T04:38:01.345 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T04:38:01.345 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi047.zgysuz.log 2023-11-15T04:38:01.346 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-mon.smithi047.log 2023-11-15T04:38:01.347 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-15T04:38:01.347 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi047.zgysuz.log: /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-mon.smithi047.log: /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-volume.log: 88.3% -- replaced with /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-15T04:38:01.351 INFO:teuthology.orchestra.run.smithi047.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T04:38:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:gzip: /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi047.zgysuz.log: file size changed while zipping 2023-11-15T04:38:01.427 INFO:teuthology.orchestra.run.smithi047.stderr: 92.7% -- replaced with /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi047.zgysuz.log.gz 2023-11-15T04:38:01.499 INFO:teuthology.orchestra.run.smithi047.stderr: 90.6% -- replaced with /var/log/ceph/4f7ad8e8-836f-11ee-8dbe-212e2dc638e7/ceph-mon.smithi047.log.gz 2023-11-15T04:38:01.500 INFO:teuthology.orchestra.run.smithi047.stderr: 2023-11-15T04:38:01.500 INFO:teuthology.orchestra.run.smithi047.stderr:real 0m0.169s 2023-11-15T04:38:01.501 INFO:teuthology.orchestra.run.smithi047.stderr:user 0m0.237s 2023-11-15T04:38:01.501 INFO:teuthology.orchestra.run.smithi047.stderr:sys 0m0.031s 2023-11-15T04:38:01.501 INFO:tasks.cephadm:Archiving logs... 2023-11-15T04:38:01.502 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354/remote/ubuntu@smithi047.front.sepia.ceph.com/log 2023-11-15T04:38:01.503 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T04:38:01.621 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354/remote/ubuntu@smithi190.front.sepia.ceph.com/log 2023-11-15T04:38:01.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T04:38:01.651 INFO:tasks.cephadm:Removing cluster... 2023-11-15T04:38:01.651 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 --force 2023-11-15T04:39:06.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4f7ad8e8-836f-11ee-8dbe-212e2dc638e7 --force 2023-11-15T04:39:06.916 INFO:tasks.cephadm:Removing cephadm ... 2023-11-15T04:39:06.916 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T04:39:06.941 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T04:39:06.960 INFO:tasks.cephadm:Teardown complete 2023-11-15T04:39:06.961 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 5: 'sudo systemctl stop ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi190' 2023-11-15T04:39:07.043 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=99113798cb9646018909e5b03f583e93 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 5: 'sudo systemctl stop ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi190' 2023-11-15T04:39:07.048 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-15T04:39:07.059 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-15T04:39:07.102 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-15T04:39:07.103 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T04:39:07.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T04:39:07.204 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-15T04:39:07.216 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-15T04:39:07.217 DEBUG:teuthology.orchestra.run.smithi047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:39:07.220 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:39:07.237 INFO:teuthology.orchestra.run.smithi190.stderr:bash: ntpq: command not found 2023-11-15T04:39:07.240 INFO:teuthology.orchestra.run.smithi047.stderr:bash: ntpq: command not found 2023-11-15T04:39:07.242 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:39:07.243 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2023-11-15T04:39:07.244 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -630us[ -630us] +/- 85ms 2023-11-15T04:39:07.244 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -7661ns[+6466ns] +/- 58ms 2023-11-15T04:39:07.245 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +538us[ +538us] +/- 64ms 2023-11-15T04:39:07.245 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:39:07.248 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:39:07.249 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2023-11-15T04:39:07.249 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 -613us[ -604us] +/- 85ms 2023-11-15T04:39:07.249 INFO:teuthology.orchestra.run.smithi047.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -22us[ -13us] +/- 58ms 2023-11-15T04:39:07.250 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +530us[ +539us] +/- 64ms 2023-11-15T04:39:07.250 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:39:07.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-15T04:39:07.261 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-15T04:39:07.262 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-15T04:39:07.297 DEBUG:teuthology.orchestra.run.smithi047:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T04:39:07.401 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T04:39:07.491 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:39:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:39:07.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:39:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:39:07.563 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-15T04:39:07.574 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-15T04:39:07.628 INFO:teuthology.task.internal:Duration was 1420.363594 seconds 2023-11-15T04:39:07.628 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-15T04:39:07.671 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-15T04:39:07.671 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T04:39:07.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T04:39:07.720 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:39:07.730 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:39:08.022 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-15T04:39:08.022 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi047.front.sepia.ceph.com 2023-11-15T04:39:08.023 DEBUG:teuthology.orchestra.run.smithi047:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T04:39:08.052 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2023-11-15T04:39:08.053 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T04:39:08.076 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-15T04:39:08.077 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T04:39:08.095 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T04:39:08.163 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-15T04:39:08.163 DEBUG:teuthology.orchestra.run.smithi047:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T04:39:08.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T04:39:08.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-15T04:39:08.266 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-15T04:39:08.266 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T04:39:08.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T04:39:08.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-15T04:39:08.337 DEBUG:teuthology.orchestra.run.smithi047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T04:39:08.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T04:39:08.371 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = core 2023-11-15T04:39:08.395 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2023-11-15T04:39:08.417 DEBUG:teuthology.orchestra.run.smithi047:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T04:39:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:39:08.480 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T04:39:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:39:08.498 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-15T04:39:08.508 INFO:teuthology.task.internal:Transferring archived files... 2023-11-15T04:39:08.509 DEBUG:teuthology.misc:Transferring archived files from smithi047:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354/remote/smithi047 2023-11-15T04:39:08.510 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T04:39:08.567 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354/remote/smithi190 2023-11-15T04:39:08.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T04:39:08.608 INFO:teuthology.task.internal:Removing archive directory... 2023-11-15T04:39:08.609 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T04:39:08.612 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T04:39:08.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-15T04:39:08.675 INFO:teuthology.task.internal:Not uploading archives. 2023-11-15T04:39:08.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-15T04:39:08.706 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-15T04:39:08.707 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T04:39:08.710 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T04:39:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 04:39 /home/ubuntu/cephtest 2023-11-15T04:39:08.730 INFO:teuthology.orchestra.run.smithi047.stdout: 525874 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 04:39 /home/ubuntu/cephtest 2023-11-15T04:39:08.732 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-15T04:39:08.757 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-15T04:39:08.859 INFO:teuthology.nuke:Checking targets against current locks 2023-11-15T04:39:08.886 DEBUG:teuthology.nuke:shortname: smithi047 2023-11-15T04:39:08.887 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T04:39:08.912 DEBUG:teuthology.nuke:shortname: smithi190 2023-11-15T04:39:08.913 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T04:39:08.916 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:07:27.471070', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG0OvF9O3QqHLKEqxWfUUJCtLydCEOZRpVt7/ApFyU5En8GBhxgd5s/biYBzUuVRYD42he6oTvsZePaLDEtDN0E='} 2023-11-15T04:39:08.941 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:07:27.469522', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyyin6RZp5iH+Tz+eE48cVroDms3tb8Lb35sH8TCFBalDeVrCdON3Gnd5FWtFMXDAuV4CVCDw3t6Jg2nQInWDs='} 2023-11-15T04:39:08.972 INFO:teuthology.orchestra.console.smithi047:Power off 2023-11-15T04:39:08.973 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T04:39:08.997 DEBUG:teuthology.orchestra.console.smithi047:power off output: Chassis Power Control: Down/Off 2023-11-15T04:39:08.998 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:39:09.013 INFO:teuthology.orchestra.console.smithi190:Power off 2023-11-15T04:39:09.014 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T04:39:09.025 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2023-11-15T04:39:09.040 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2023-11-15T04:39:09.040 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:39:09.067 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-11-15T04:39:13.027 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:39:13.069 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:39:13.138 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2023-11-15T04:39:13.182 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-11-15T04:39:17.142 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:39:17.184 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:39:17.253 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is off 2023-11-15T04:39:17.297 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2023-11-15T04:39:17.355 INFO:teuthology.orchestra.console.smithi047:Power off completed 2023-11-15T04:39:17.398 INFO:teuthology.orchestra.console.smithi190:Power off completed 2023-11-15T04:39:17.505 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1420.3635940551758 failure_reason: 'Command failed on smithi190 with status 5: ''sudo systemctl stop ceph-4f7ad8e8-836f-11ee-8dbe-212e2dc638e7@mon.smithi190''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=99113798cb9646018909e5b03f583e93 status: fail success: false 2023-11-15T04:39:17.505 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T04:39:17.591 INFO:teuthology.run:FAIL